Spark 2.4升级至3.2.1后集群模式长时任务报错求助

问题描述 #

将Spark版本从2.4升级到3.2.1后，部分长时间运行的任务出现如下错误，已调优所有Spark配置但仍无法解决。此前了解到该错误常出现在客户端模式下的数据密集型任务，但所有任务均运行在集群模式。

错误日志：

24/02/15 14:14:39 INFO Client: Application report for application_1707932596261_1053 (state: FINISHED)
24/02/15 14:14:39 INFO Client: 
     client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
     diagnostics: Shutdown hook called before final status was reported.
     ApplicationMaster host: qab01wn0.sbn1010026024qa.com
     ApplicationMaster RPC port: 23671
     queue: default
     start time: 1708006450401
     final status: FAILED
     tracking URL: http://qaserver.sbn1010026024qa:8088/proxy/application_1707932596261_1053/
     user: spark
24/02/15 14:14:39 ERROR Client: Application diagnostics message: Shutdown hook called before final status was reported.
Exception in thread "main" org.apache.spark.SparkException: Application application_1707932596261_1053 finished with failed status
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1283)
    at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1677)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
24/02/15 14:14:39 INFO ShutdownHookManager: Shutdown hook called
24/02/15 14:14:39 INFO ShutdownHookManager: Deleting directory /tmp/spark-98e785bd-9a24-4ac2-99f6-00794d74a773
24/02/15 14:14:39 INFO ShutdownHookManager: Deleting directory /tmp/spark-9a84a0b2-8196-4bc6-a123-ca53ca30a916

解决建议 #

• 调整YARN客户端超时配置：集群模式下提交任务的Client进程可能因等待AM状态超时被终止，触发ShutdownHook。可尝试增加spark.yarn.submit.waitAppCompletion的超时逻辑（确保设置为true），调整YARN端的yarn.client.application-client-protocol.poll-interval-ms延长轮询间隔，同时检查集群是否存在对提交进程的超时限制（如OS进程存活时间、资源管理工具规则）。
• 排查Spark与YARN兼容性：Spark 3.2.1对YARN交互逻辑有调整，可能存在AM与Client状态同步延迟。建议升级YARN到兼容版本（官方推荐Hadoop 3.2.x及以上），启用spark.yarn.am.waitTime延长AM等待状态同步的时间。
• 优化任务执行逻辑：拆分长任务为多个小阶段，减少单任务运行时长；排查数据倾斜问题，避免部分Task拖慢整体进度；调整spark.executor.instances、spark.executor.memory等资源参数，提升任务执行效率。
• 调整ShutdownHook相关配置：启用外部Shuffle服务（spark.shuffle.service.enabled=true），减少Driver端资源占用；检查是否有自定义ShutdownHook逻辑与Spark 3.x的钩子逻辑冲突。

内容的提问来源于stack exchange，提问作者user13906258