Spark报错FetchFailedException的成因及解决方法咨询

错误日志 #

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: ShuffleMapStage 18 ($anonfun$withThreadLocalCaptured$5 at LexicalThreadLocal.scala:63) has failed the maximum allowable number of times: 4. Most recent failure reason:
org.apache.spark.shuffle.FetchFailedException. 


..............
Caused by: java.io.IOException: Connection from /XX.XXX.XX.XXX:4048 closed

问题概况 #

作业占用大量资源，运行6小时后失败，暂未进行任何调整。

测试调整计划 #

• 减少作业的分区数
• 移除自定义Spark配置中的spark.driver.maxResultSize 2g参数
• 移除自定义Spark配置中的spark.driver.memory 80g参数

内容的提问来源于stack exchange，提问作者waliadee