Spark History-Server（3.2.1）部分应用日志不显示问题求助

问题现象 #

Spark History Server（版本3.2.1，搭配Hadoop 3.1.1）随机出现部分应用日志无法展示的情况：

• 所有应用日志均存储在HDFS的同一spark-events路径下
• 重启History Server后，之前不显示的日志可正常展示
• 运行一段时间后，新的一批应用日志又会出现无法显示的问题，无明确规律

相关错误日志 #

23/08/16 20:59:40 INFO FsHistoryProvider: Parsing hdfs://stagehdfs/spark3-history/application_1678437047884_599309_1.inprogress for listing data...
23/08/16 20:59:40 INFO FsHistoryProvider: Parsing hdfs://stagehdfs/spark3-history/application_1678437047884_599304_1 for listing data...
23/08/16 20:59:40 INFO FsHistoryProvider: Looking for end event; skipping 112931832 bytes from hdfs://stagehdfs/spark3-history/application_1678437047884_599304_1...
23/08/16 20:59:40 INFO FsHistoryProvider: Finished parsing hdfs://stagehdfs/spark3-history/application_1678437047884_599304_1
23/08/16 20:59:40 INFO FsHistoryProvider: Finished parsing hdfs://stagehdfs/spark3-history/application_1678437047884_599309_1.inprogress
23/08/16 21:00:47 INFO FsHistoryProvider: Parsing hdfs://stagehdfs/spark3-history/application_1678437047884_599319_1.inprogress for listing data...
23/08/16 21:00:47 INFO FsHistoryProvider: Parsing hdfs://stagehdfs/spark3-history/application_1678437047884_599318_1.inprogress for listing data...
23/08/16 21:00:47 INFO FsHistoryProvider: Parsing hdfs://stagehdfs/spark3-history/application_1678437047884_599317_1.inprogress for listing data...
23/08/16 21:00:47 INFO FsHistoryProvider: Finished parsing hdfs://stagehdfs/spark3-history/application_1678437047884_599318_1.inprogress
23/08/16 21:00:47 INFO FsHistoryProvider: Finished parsing hdfs://stagehdfs/spark3-history/application_1678437047884_599319_1.inprogress
23/08/16 21:00:47 INFO FsHistoryProvider: Finished parsing hdfs://stagehdfs/spark3-history/application_1678437047884_599317_1.inprogress
23/08/16 21:02:24 INFO FsHistoryProvider: Parsing hdfs://stagehdfs/spark3-history/application_1678437047884_599319_1 for listing data...
23/08/16 21:02:24 INFO FsHistoryProvider: Looking for end event; skipping 82041452 bytes from hdfs://stagehdfs/spark3-history/application_1678437047884_599319_1...
23/08/16 21:02:24 INFO FsHistoryProvider: Finished parsing hdfs://stagehdfs/spark3-history/application_1678437047884_599319_1
23/08/16 21:02:24 ERROR Utils: Uncaught exception in thread log-replay-executor-7
java.util.NoSuchElementException
        at org.apache.spark.util.kvstore.InMemoryStore.read(InMemoryStore.java:85)
        at org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$checkAndCleanLog$3(FsHistoryProvider.scala:927)
        at scala.Option.foreach(Option.scala:407)
        at org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$checkAndCleanLog$1(FsHistoryProvider.scala:926)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.util.Utils$.tryLog(Utils.scala:2032)
        at org.apache.spark.deploy.history.FsHistoryProvider.checkAndCleanLog(FsHistoryProvider.scala:916)
        at org.apache.spark.deploy.history.FsHistoryProvider.mergeApplicationListing(FsHistoryProvider.scala:712)
        at org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$checkForLogs$15(FsHistoryProvider.scala:576)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
23/08/16 21:02:34 INFO FsHistoryProvider: Parsing hdfs://stagehdfs/spark3-history/application_1678437047884_599318_1 for listing data...
23/08/16 21:02:34 INFO FsHistoryProvider: Looking for end event; skipping 82754288 bytes from hdfs://stagehdfs/spark3-history/application_1678437047884_599318_1...

问题分析 #

日志中的java.util.NoSuchElementException来自InMemoryStore.read方法，说明Spark History Server使用的内存KV存储在并发访问时出现了数据丢失或找不到的情况。具体是日志回放线程（log-replay-executor）在处理应用日志时，尝试读取内存中已被清理线程移除的记录，导致解析中断，进而该应用日志无法被History Server加载展示。

解决方案 #

• 升级Spark版本到3.2.2及以上：官方在Spark 3.2.2中修复了多个FsHistoryProvider并发访问的问题，包括InMemoryStore的线程安全缺陷，升级后可直接解决该异常。
• 切换到FileSystemBackedStore存储：在spark-defaults.conf中配置spark.history.store.path为本地磁盘或HDFS路径，将历史数据从内存存储改为文件系统存储，避免内存并发冲突：

  spark.history.store.path=/path/to/local/dir  # 或 hdfs://path/to/hdfs/dir
  

• 调整History Server线程配置：增大日志回放线程池大小，减少并发冲突概率：

  spark.history.fs.threadPool.size=4
  spark.history.fs.update.interval=15s  # 延长扫描间隔，降低线程触发频率
  

• 检查HDFS日志文件完整性：对无法显示的应用日志文件执行HDFS检查，确认文件未损坏：

  hdfs fsck hdfs://stagehdfs/spark3-history/application_1678437047884_599319_1
  

  若文件损坏且对应应用已结束，可删除该损坏文件，History Server下次扫描时会重新解析其他正常文件。

内容的提问来源于stack exchange，提问作者Manju N