Spark Driver与Executor通信引发OOM的排查方法咨询

错误堆栈 #

java.lang.OutOfMemoryError: Java heap space
  at java.util.Arrays.copyOf(Arrays.java:3332)
  at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
  at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
  at java.lang.StringBuilder.append(StringBuilder.java:141)
  at java.lang.StringBuilder.append(StringBuilder.java:136)
  at java.util.AbstractCollection.toString(AbstractCollection.java:462)
  at java.util.Collections$UnmodifiableCollection.toString(Collections.java:1037)
  at org.apache.spark.util.JsonProtocol$.accumValueToJson(JsonProtocol.scala:469)
  at org.apache.spark.util.JsonProtocol$.$anonfun$accumulableInfoToJson$7(JsonProtocol.scala:428)
  at org.apache.spark.util.JsonProtocol$$$Lambda$4808/1475607995.apply(Unknown Source)
  at scala.Option.map(Option.scala:230)
  at org.apache.spark.util.JsonProtocol$.accumulableInfoToJson(JsonProtocol.scala:428)
  at org.apache.spark.util.JsonProtocol$.$anonfun$accumulablesToJson$4(JsonProtocol.scala:420)
  at org.apache.spark.util.JsonProtocol$$$Lambda$4204/1507248734.apply(Unknown Source)
  at scala.collection.immutable.List.map(List.scala:293)
  at org.apache.spark.util.JsonProtocol$.accumulablesToJson(JsonProtocol.scala:420)
  at org.apache.spark.util.JsonProtocol$.taskInfoToJson(JsonProtocol.scala:412)
  at org.apache.spark.util.JsonProtocol$.taskEndToJson(JsonProtocol.scala:171)
  at org.apache.spark.util.JsonProtocol$.sparkEventToJson(JsonProtocol.scala:81)
  at org.apache.spark.deploy.history.rpc.app.RpcAppEvent$.encodeEvent(RpcAppEvent.scala:77)
  at org.apache.spark.deploy.history.rpc.app.RpcAppEvent$.apply(RpcAppEvent.scala:62)
  at org.apache.spark.deploy.history.rpc.app.RpcAppEventQueue.enqueue(RpcAppEventQueue.scala:86)
  at org.apache.spark.deploy.history.rpc.app.RpcAppListener.addToEventQueue(RpcAppListener.scala:75)
  at org.apache.spark.deploy.history.rpc.app.RpcAppListener.onEvent(RpcAppListener.scala:83)
  at org.apache.spark.deploy.history.rpc.util.SingleHandlerListener.onTaskEnd(SingleHandlerListener.scala:48)
  at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:45)
  at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
  at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
  at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
  at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:120)
  at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:104)
  at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:127)

Spark配置 #

spark.dynamicAllocation.maxExecutors=9
spark.executor.cores=2
spark.dynamicAllocation.minExecutors=1
spark.dynamicAllocation.initialExecutors=1
spark.executor.instances=8
spark.shuffle.service.client.class=org.apache.spark.network.shuffle.ExternalShuffleClient
spark.driver.maxResultSize=1g
spark.sql.files.maxPartitionBytes=1073741824
spark.sql.join.preferSortMergeJoin=true
spark.dynamicAllocation.executorIdleTimeout=300s
spark.executor.extraJavaOptions=-Detwlogger.component=sparkexecutor -DlogFilter.filename=SparkLogFilters.xml -DpatternGroup.filename=SparkPatternGroups.xml -Dlog4jspark.root.logger=INFO,console,RFA,ETW,Anonymizer -Dlog4jspark.log.dir=/var/log/sparkapp/${user.name} -Dlog4jspark.log.file=sparkexecutor.log -Dlog4j2.configurationFile=file:/usr/hdp/current/spark3-client/conf/executor-log4j2.properties -Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl -XX:+UseParallelGC -XX:+UseParallelOldGC
spark.driver.cores=1
spark.driver.memory=**2g**
spark.executor.memory=9g
spark.driver.memoryOverhead=**384**
spark.executor.memoryOverhead=921

场景与问题 #

• 集群配置9个Executor，任务总数919个，代码在df.rdd.foreachPartition中执行，使用了int和string类型的累加器（已控制string累加器不会无限增长）。
• 每个任务完成后日志显示，单任务向Driver发送的结果约60000-62000字节，无异常值：

INFO Executor [Executor task launch worker for task 258.0 in stage 59.0 (TID 4666)]: Finished task 258.0 in stage 59.0 (TID 4666). 61394 bytes result sent to driver

• 任务完成约270/919时，Spark触发OOM错误。

疑问与需求 #

从错误堆栈看，推测OOM发生在Executor向Driver通信过程中，Driver在序列化累积信息为JSON时出现内存不足。但270个任务的总结果约15MB，理论上不应导致2G内存的Driver OOM，怀疑是内部存在多次内存复制导致内存占用激增。

虽然直接增加Driver内存可以解决问题，但希望了解进一步的排查方法，以验证上述理论——即OOM确实由Executor到Driver的通信环节引发。

内容的提问来源于stack exchange，提问作者Igor Railean