Spark-submit提交Yarn任务时Worker无法解析调用方容器主机名问询

问题背景 #

使用spark-submit --master yarn foo.py从Docker容器提交Spark任务到Yarn集群时，Worker节点日志出现java.net.UnknownHostException: e4f76e66ba41（e4f76e66ba41为Docker容器主机名），无法解析该主机名导致连接失败。

提交任务日志片段 #

sara@e4f76e66ba41:~$ spark-submit --master yarn foo.py
23/10/11 20:41:00 INFO SparkContext: Running Spark version 3.5.0
23/10/11 20:41:00 INFO SparkContext: OS info Linux, 5.10.0-25-cloud-amd64, amd64
23/10/11 20:41:00 INFO SparkContext: Java version 17.0.8.1
23/10/11 20:41:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/10/11 20:41:00 INFO ResourceUtils: ==============================================================
23/10/11 20:41:00 INFO ResourceUtils: No custom resources configured for spark.driver.
23/10/11 20:41:00 INFO ResourceUtils: ==============================================================
23/10/11 20:41:00 INFO SparkContext: Submitted application: foo.py
23/10/11 20:41:00 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name:
.........
Traceback (most recent call last):
File "/home/sara/foo.py", line 8, in 
spark = SparkSession.builder.getOrCreate()
^^^23/10/11 20:41:20 INFO MemoryStore: MemoryStore cleared
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/session.py", line 497, in getOrCreate
23/10/11 20:41:20 INFO BlockManager: BlockManager stopped
File "/opt/conda/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/context.py", line 515, in getOrCreate
File "/opt/conda/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/context.py", line 203, in __init__
File "/opt/conda/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/context.py", line 296, in _do_init
File "/opt/conda/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/context.py", line 421, in _initialize_context
File "/opt/conda/lib/python3.11/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1587, in __call__
File "/opt/conda/lib/python3.11/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value

Worker节点错误日志片段 #

at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:933)
    at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:965)
    at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
Caused by: java.io.IOException: Failed to connect to e4f76e66ba41:41395
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:294)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:214)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:226)
    at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.UnknownHostException: e4f76e66ba41
    at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1386)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1307)
    at java.base/java.net.InetAddress.getByName(InetAddress.java:1257)
    at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156)
    at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153)
    at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41)
    at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61)
    at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53)
    at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55)

问题解答 #

1. 现象产生的原因 #

默认情况下，Spark以client模式提交到Yarn时，Driver程序会运行在提交任务的Docker容器上。Worker节点的Executor必须与Driver建立RPC通信，完成任务指令获取、执行状态汇报、计算数据传输等核心流程。由于Docker容器的主机名未在Yarn集群的DNS系统中注册，Worker节点无法解析该主机名，导致连接失败。

2. 是否属于Spark的预期行为 #

这是Spark的预期行为。client模式下Driver依赖提交节点的网络可达性，Executor必须反向连接到Driver；若使用cluster模式，Driver会运行在Yarn集群内部，Worker仅需与集群内的Driver通信，不会出现该问题。

3. Worker节点是否需要访问提交任务的调用方 #

分两种场景：

• client模式：必须访问。Driver运行在提交节点上，Executor与Driver的双向通信是任务执行的基础。
• cluster模式：不需要。Driver由Yarn托管在集群内部，提交节点在任务提交完成后可断开连接，Worker仅与集群内的Driver交互。

设计逻辑说明 #

这种模式划分是为了适配不同场景：

• client模式适合调试，Driver在本地运行便于查看日志和实时调试；
• cluster模式适合生产环境，Driver由Yarn管理，避免外部节点的网络依赖，提交节点无需长期在线。

解决办法 #

可通过以下方式解决该问题：

• 切换到cluster模式：提交任务时添加--deploy-mode cluster参数，让Driver运行在Yarn集群内部，命令示例：

  spark-submit --master yarn --deploy-mode cluster foo.py
  

• 配置DNS解析：在所有Worker节点的/etc/hosts文件中添加Docker容器的主机名与IP映射，或在集群DNS服务器注册该主机名，确保Worker能解析e4f76e66ba41。
• 指定Driver的外部访问地址：提交任务时设置spark.driver.host为Docker容器的可访问IP，替代自动生成的主机名，命令示例：

  spark-submit --master yarn --conf spark.driver.host=192.168.1.100 foo.py
  

内容的提问来源于stack exchange，提问作者ljmask