Docker化环境中Spark Worker无法解析应用主机问题排查

问题场景 #

我将Spark以Docker微服务形式部署，Spark应用能提交任务到Worker节点，但Worker无法反向连接应用，抛出UnknownHostException。Worker尝试通过容器ID（比如658e5d214a60）与应用通信，但该容器ID无法解析为容器IP。本地Linux机器用docker-compose运行正常，但在AWS EC2容器中运行失败，报错信息如下：

Exception in thread "main" java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1713) at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:64) at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188) at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:293) at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala) Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205) at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75) at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101) at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:201) at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:65) at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:64) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) ... 4 more Caused by: java.io.IOException: Failed to connect to 658e5d214a60:36335 at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245) at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187) at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198) at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194) at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.UnknownHostException: 658e5d214a60 at java.net.InetAddress.getAllByName0(InetAddress.java:1259) at java.net.InetAddress.getAllByName(InetAddress.java:1171) at java.net.InetAddress.getAllByName(InetAddress.java:1105) at java.net.InetAddress.getByName(InetAddress.java:1055) at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:146) at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:143) at java.security.AccessController.doPrivileged(Native Method) at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:143) at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:43) at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:63) at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:55) at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:57) at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:32) at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:108) at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:208) at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:49) at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:188) at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:174) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:978) at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:512) at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) at io.netty.util.concurrent.SingleThrea

问题根源 #

Spark Driver启动时会默认把自己的容器ID作为通信地址广播给Worker，但在AWS EC2的Docker环境里，Worker节点的DNS服务没办法解析这个容器ID（本地Docker的DNS默认支持容器ID解析，但AWS上的配置可能有差异），导致Worker找不到Driver的IP。

解决方案 #

1. 用Docker Compose服务名指定Driver地址（最推荐）

如果你是用docker-compose编排集群，直接把Driver的服务名作为广播地址就行——Compose默认创建的网络里，服务名可以被所有容器解析，不需要依赖不稳定的容器ID。

提交Spark应用时添加这些配置：

spark-submit \
  --conf spark.driver.host=spark-driver \  # 替换成你compose里Driver服务的名称
  --conf spark.driver.bindAddress=0.0.0.0 \  # 让Driver监听所有接口
  --conf spark.driver.port=4040 \  # 固定端口，方便AWS安全组配置
  # 你的应用jar包和参数...

也可以在Spark的spark-defaults.conf里持久化这些配置，避免每次提交都加：

spark.driver.host=spark-driver
spark.driver.bindAddress=0.0.0.0
spark.driver.port=4040

2. 绑定Driver到容器固定IP（备选）

如果不想用服务名，也可以指定Driver容器的IP，但这种方式不够灵活（容器重启后IP可能变化）。你可以先获取Driver容器的IP，再提交应用：

# 获取Driver容器的IP地址
DRIVER_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <你的Driver容器名称>)

# 提交应用时指定IP
spark-submit \
  --conf spark.driver.host=$DRIVER_IP \
  --conf spark.driver.bindAddress=0.0.0.0 \
  # 其他应用参数...

3. 检查AWS EC2安全组配置

AWS的安全组会限制流量，必须确保允许以下端口的入站/出站：

• Spark Master端口：7077
• Driver固定端口：比如刚才设置的4040
• Executor端口范围：默认是4041及以上，或者通过spark.executor.port固定一个范围，方便安全组规则配置

4. 确认Docker网络一致性

确保所有Spark容器（Master、Worker、Driver）都在同一个Docker网络里——docker-compose默认会创建专属网络，只要你没手动修改网络配置，这一步通常没问题。如果是手动创建网络，要确认所有容器都加入了该网络。

验证方法 #

配置完成后重启集群，提交应用后查看Worker日志，如果不再出现容器ID解析错误，而是成功连接到你指定的服务名或IP，就说明问题解决了。

内容的提问来源于stack exchange，提问作者alk453