K8s集群中Spark Worker杀死应用问题求助

问题背景 #

在Kubernetes集群中运行Spark应用，集群包含1个Master Pod和3个Worker Pod。提交任务后，某Worker Pod会立即杀死应用导致任务失败。该Worker的启动命令与其他正常Worker不符，同时Master日志出现Failed to send one-way RPC和NoRouteToHost异常。

相关日志 #

Worker Pod日志 #

23/07/12 12:15:13 INFO Worker: Successfully registered with master spark://sparkmaster-0.sparkmaster-hs.default.svc.cluster.local:7077
23/07/12 12:19:55 INFO Worker: Asked to launch executor app-20230712121955-0000/0 for Job 1003597
23/07/12 12:19:55 INFO SecurityManager: Changing view acls groups to: 
23/07/12 12:19:55 INFO SecurityManager: Changing modify acls groups to: 
23/07/12 12:19:55 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(); groups with view permissions: Set(); users  with modify permissions: Set(); groups with modify permissions: Set()
23/07/12 12:19:55 INFO ExecutorRunner: Launch command: "/usr/java/jdk-11/bin/java" "-cp" "/u01/spark/conf/:/u01/spark/jars/*" "-Xmx51200M" "-Dspark.driver.port=33143" "-Dspark.rpc.askTimeout=10s" "-XX:+UseG1GC" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@sparkworker-1.sparkworker-hs.default.svc.cluster.local:33143" "--executor-id" "0" "--hostname" "10.244.1.239" "--cores" "1" "--app-id" "app-20230712121955-0000" "--worker-url" "spark://Worker@10.244.1.239:37193"
23/07/12 12:20:20 INFO Worker: Asked to kill executor app-20230712121955-0000/0
23/07/12 12:20:20 INFO ExecutorRunner: Runner thread for executor app-20230712121955-0000/0 interrupted
23/07/12 12:20:20 INFO ExecutorRunner: Killing process!
23/07/12 12:20:21 INFO Worker: Executor app-20230712121955-0000/0 finished with state KILLED exitStatus 143
23/07/12 12:20:21 INFO ExternalShuffleBlockResolver: Clean up non-shuffle files associated with the finished executor 0

Master Pod日志 #

23/07/12 12:15:13 INFO Master: Registering worker 10.244.1.239:37193 with 1 cores, 57.6 GB RAM
23/07/12 12:15:40 INFO Master: 10.244.2.12:38819 got disassociated, removing it.
23/07/12 12:15:40 INFO Master: Removing worker worker-20230712101331-10.244.2.12-38819 on 10.244.2.12:38819
23/07/12 12:15:40 INFO Master: Telling app of lost worker: worker-20230712101331-10.244.2.12-38819
23/07/12 12:15:40 INFO Master: 10.244.2.12:38819 got disassociated, removing it.
23/07/12 12:15:57 INFO Master: Registering worker 10.244.1.240:33237 with 1 cores, 57.6 GB RAM
23/07/12 12:16:24 INFO Master: 10.244.2.140:46723 got disassociated, removing it.
23/07/12 12:16:24 INFO Master: Removing worker worker-20230712101508-10.244.2.140-46723 on 10.244.2.140:46723
23/07/12 12:16:24 INFO Master: Telling app of lost worker: worker-20230712101508-10.244.2.140-46723
23/07/12 12:16:24 INFO Master: 10.244.2.140:46723 got disassociated, removing it.
23/07/12 12:17:48 INFO Master: Registering worker 10.244.2.141:41993 with 1 cores, 57.6 GB RAM
23/07/12 12:19:50 INFO Master: Driver submitted org.apache.spark.deploy.worker.DriverWrapper
23/07/12 12:19:50 INFO Master: Launching driver driver-20230712121950-0000 on worker worker-20230712121556-10.244.1.240-33237
23/07/12 12:19:55 INFO Master: Registering app Job 1003597
23/07/12 12:19:55 INFO Master: Registered app Job 1003597 with ID app-20230712121955-0000
23/07/12 12:19:55 INFO Master: Launching executor app-20230712121955-0000/0 on worker worker-20230712121513-10.244.1.239-37193
23/07/12 12:19:55 INFO Master: 10.244.1.237:36488 got disassociated, removing it.
23/07/12 12:19:55 INFO Master: 10.244.1.237:35713 got disassociated, removing it.
23/07/12 12:20:20 INFO Master: Received unregister request from application app-20230712121955-0000
23/07/12 12:20:20 INFO Master: Removing app app-20230712121955-0000
23/07/12 12:20:20 INFO TransportClientFactory: Found inactive connection to /10.244.2.12:38819, creating a new one.
23/07/12 12:20:20 INFO TransportClientFactory: Found inactive connection to /10.244.2.140:46723, creating a new one.
23/07/12 12:20:20 INFO TransportClientFactory: Found inactive connection to /10.244.2.10:37231, creating a new one.
23/07/12 12:20:20 INFO Master: 10.244.1.240:46296 got disassociated, removing it.
23/07/12 12:20:20 INFO Master: sparkworker-1.sparkworker-hs.default.svc.cluster.local:33143 got disassociated, removing it.
23/07/12 12:20:20 INFO Master: Removing driver: driver-20230712121950-0000
23/07/12 12:20:21 WARN Master: Got status update for unknown executor app-20230712121955-0000/0
23/07/12 12:20:23 WARN OneWayOutboxMessage: Failed to send one-way RPC.
java.io.IOException: Failed to connect to /10.244.2.140:46723
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:244) ~[spark-network-common_2.11-2.4.5.jar:2.4.5]
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:186) ~[spark-network-common_2.11-2.4.5.jar:2.4.5]
    at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198) ~[spark-core_2.11-2.4.5.jar:2.4.5]
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194) [spark-core_2.11-2.4.5.jar:2.4.5]
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190) [spark-core_2.11-2.4.5.jar:2.4.5]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: /10.244.2.140:46723
Caused by: java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]

关键异常信息 #

23/07/12 12:20:23 WARN OneWayOutboxMessage: Failed to send one-way RPC.
java.io.IOException: Failed to connect to /10.244.2.140:46723

问题分析 #

1. Executor被主动杀死的原因：从Worker日志可见，Executor是被Master下达指令杀死的（Asked to kill executor），而非自身崩溃。
2. NoRouteToHost异常含义：该异常表示Master无法与指定IP（10.244.2.140）的Worker Pod建立网络连接，原因可能是：
   • 该Worker Pod已离线或被销毁（Master日志显示该Worker在12:16:24已被标记为disassociated并移除）
   • Kubernetes网络策略限制了Master与该Worker的通信
   • Node节点之间存在网络不通问题
3. Worker启动命令不符的影响：异常Worker的启动参数可能存在配置错误（如内存分配、Driver地址指向错误），导致Executor启动后无法与Driver/Master正常通信，触发Master的清理逻辑。

解决步骤 #

1. 修复异常Worker的启动命令 #

• 对比正常Worker的启动参数，修正异常Worker的JVM参数、Driver URL、Spark配置项。重点检查：
  • --driver-url是否指向正确的Driver Pod地址
  • -Xmx内存分配是否符合集群资源限制
  • Spark RPC相关配置（如spark.rpc.askTimeout）是否统一

2. 排查网络连通性问题 #

• 在Master Pod中执行ping 10.244.2.140和telnet 10.244.2.140 46723，确认网络是否可达
• 检查Kubernetes网络策略，确保Spark Master与Worker之间的通信端口（默认7077、Executor RPC端口）未被阻止
• 检查Worker所在Node节点的网络状态，确认Node之间的Pod网络正常

3. 清理失效Worker并重建 #

• 删除已标记为disassociated的Worker Pod（如10.244.2.12、10.244.2.140），等待StatefulSet或Deployment重建健康的Worker
• 重启Spark Master Pod，确保Master重新发现所有健康Worker

4. 验证任务执行 #

• 提交测试Spark任务，观察Executor是否正常启动，任务是否完成
• 监控Master和Worker日志，确认无NoRouteToHost或Executor被杀死的异常

内容的提问来源于stack exchange，提问作者Eli Johnes