K8s集群Spark Executor无法连接Driver报UnknownHostException

问题现象 #

在K8s集群内通过Pod执行spark-submit提交集群模式Spark任务，提交脚本如下：

/opt/spark/bin/spark-submit
--master k8s://someCluster
--deploy-mode cluster
--name someName
--class some.class
--conf spark.driver.userClassPathFirst=true         
--conf spark.kubernetes.namespace=someNamespace
--conf spark.kubernetes.container.image=someImage
--conf spark.kubernetes.container.image.pullSecrets=image-pull-secret
--conf spark.kubernetes.container.image.pullPolicy=Always
--conf spark.kubernetes.authenticate.submission.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token
--conf spark.kubernetes.authenticate.driver.serviceAccountName=someServiceAccount
--conf spark.driver.port=7078
--conf spark.blockManager.port=7079
local:////someApp.jar

提交后出现两个关联异常：

• Driver Pod正常启动，自动创建的Driver Service已配置7078、7079、4040端口规则，标签选择器和Driver Pod标签完全匹配，但Service始终无可用Endpoints
• Executor Pod启动后立即失败，抛出java.net.UnknownHostException: <driverName>.<namespace>.svc错误，无法解析Driver服务域名连接7078端口，核心错误栈如下：

Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1748)
        at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
        at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:283)
        at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:272)
        at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:302)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
        at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
        at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$3(CoarseGrainedExecutorBackend.scala:303)
        at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
        at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
        at scala.collection.immutable.Range.foreach(Range.scala:158)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
        at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$1(CoarseGrainedExecutorBackend.scala:301)
        at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
        at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        ... 4 more
Caused by: java.io.IOException: Failed to connect to drivername-svc.namespace.svc:7078
        at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:253)
        at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:195)
        at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
        at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
        at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.UnknownHostException: drivername-svc.namespace.svc
        at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
        at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
        at java.base/java.net.InetAddress.getByName(InetAddress.java:1248)
        at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156)
        at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153)
        at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41)
        at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61)
        at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53)
        at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55)
        at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31)
        at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106)
        at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:200)
        at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:46)
        at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:180)
        at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:166)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
        at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
        at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:984)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:504)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:417)
        at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:474)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

已配置网络策略放通7078、7079端口出入站流量，问题仍存在。

根因说明 #

Service无Endpoints是域名解析失败的直接原因：K8s CoreDNS不会为没有健康后端端点的ClusterIP Service生成DNS解析记录，Executor访问Driver域名时自然返回UnknownHostException。
标签匹配但无Endpoints，唯一可能是Driver Pod未通过就绪检查，未被Endpoints控制器纳入后端。Spark on K8s会自动为Driver Pod配置就绪探针，检测7078、4040端口连通性，探针失败的Pod永远不会被关联到Service后端。
结合提交参数判断，最高概率诱因是开启了spark.driver.userClassPathFirst=true：该配置会反转类加载优先级，让用户Jar中的类优先于Spark内置类加载，极易冲掉探针、RPC模块的依赖类版本，导致探针检测逻辑无法正常执行，Pod持续处于NotReady状态。
次高频诱因是Driver端口绑定异常：手动指定固定端口后，Driver默认可能绑定127.0.0.1回环地址而非Pod IP，导致探针检测端口时连通失败。

修复步骤 #

按优先级依次操作：

• 移除spark.driver.userClassPathFirst=true配置
  该参数是Spark on K8s场景下的高频坑点，类优先级反转会引发各类依赖冲突。如果确实需要解决类冲突，用spark.driver.extraClassPath、spark.executor.extraClassPath单独指定需要优先加载的Jar路径即可，不要全局开启类加载优先级反转。
• 补充端口绑定配置，强制Driver监听所有网卡
  在提交参数中新增以下两项配置，避免Driver绑定回环地址导致探针、Executor连接失败：

  --conf spark.driver.bindAddress=0.0.0.0
  --conf spark.driver.host=$(POD_IP)
  

  其中POD_IP通过Downward API在提交Pod中注入为环境变量即可，值为Pod自身IP，Spark会自动将该值同步给Driver和Executor。
• 临时关闭就绪探针快速验证
  如果上述修改后仍无Endpoints，新增配置临时关闭Driver自动就绪探针，快速定位问题：

  --conf spark.kubernetes.driver.pod.readinessProbe.enabled=false
  

  关闭后只要Driver Pod处于Running状态就会被加入Service后端，如果此时Executor能正常连接，可确认是探针检测逻辑异常，后续再针对性调整探针的检测延迟、超时参数即可。
• 校验ServiceAccount权限
  确认配置的someServiceAccount拥有当前Namespace下Pod、Endpoints、Service的读写权限，避免Spark提交客户端无法正常同步Pod状态到Service端点。

内容的提问来源于stack exchange，提问作者iAmHereForHelp