PySpark读取Hive表遇GC Overhead Limit Exceed错误求配置方案

读取Hive表db.emp_table（筛选条件month=202206）时出现GC Overhead Limit Exceed错误，已调整spark.executor.memory和spark.driver.memory参数但问题未解决，以下是优化后的Spark配置及关键调整说明。

优化后的SparkSession配置 #

spark = SparkSession\
.builder\
.appName ("test")\
.config("spark.master","yarn")\
.config("spark.submit.deployMode","client")\
# 内存基础配置
.config("spark.executor.memory","20g")\
.config("spark.driver.memory","25g")\
.config("spark.driver.maxResultSize","20g")  # 根据实际结果大小调整，避免内存浪费
# GC策略优化（G1GC更适配大内存场景）
.config("spark.driver.extraJavaOptions","-XX:MaxDirectMemorySize=16g -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=40 -XX:MaxGCPauseMillis=200")\
.config("spark.executor.extraJavaOptions","-XX:MaxDirectMemorySize=16g -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=40 -XX:MaxGCPauseMillis=200")\
# 动态资源分配（避免与固定实例数冲突）
.config("spark.dynamicAllocation.enabled","true")\
.config("spark.dynamicAllocation.minExecutors","10")\
.config("spark.dynamicAllocation.maxExecutors","40")\
.config("spark.dynamicAllocation.initialExecutors","20")\
# 并行度与分区控制
.config("spark.sql.shuffle.partitions","1000")\
.config("spark.default.parallelism","2000")\
.config("spark.sql.files.maxPartitionBytes","128m")  # 控制读取时的单分区数据量
# 其他核心配置
.config("spark.network.timeout","300s")\
.config("spark.speculation","true")\
.config("spark.sql.parquet.binaryAsString","true")\
.config("spark.sql.broadcastTimeout","300s")\
.config("spark.core.connection.ack.wait.timeout","60s")\
.config("spark.shuffle.compress", "true")\
.config("spark.shuffle.spill.compress", "true")\
.config("spark.shuffle.io.maxRetries","3")\
.config("spark.shuffle.registration.timeout", "10s")\
.config("spark.shuffle.registration.maxAttempts","5")\
.config("spark.yarn.queue", "root-digviz")\
.config("spark.memory.fraction","0.8")\
.config("spark.memory.storageFraction","0.3")\
.config("spark.yarn.dist.files","/opt/mapr/hive/hive/conf/hive-site.xml")\
.config("spark.sql.catalogImplementation","hive")\
.config("spark.jars","/opt/pltf/cloak/lib/cloak-spark.jar,/app/spy/hive-contrib.jar")\
.config("spark.yarn.archive","/opt/pltf/cloak/lib/spark-jars.zip")\
.enableHiveSupport()\
.getOrCreate()

sc = spark.sparkContext

# 优化读取逻辑：只读取业务需要的字段，避免全量加载
df = spark.sql("select col1, col2, col3 from db.emp_table where month=202206")

关键调整说明 #

• GC策略替换：将CMS GC替换为G1GC，G1GC更适合大内存场景，可精准控制GC停顿时间；通过InitiatingHeapOccupancyPercent调整堆占用触发GC的阈值，减少频繁GC的概率。
• 内存参数修正：将MaxDirectMemorySize调整为合理值（16g），避免内存过度分配导致系统资源耗尽；spark.driver.maxResultSize根据实际结果大小设置，避免driver内存溢出。
• 并行度优化：统一spark.sql.shuffle.partitions配置，避免重复设置冲突；调整spark.default.parallelism与集群资源匹配，避免任务并行度过高或过低；通过spark.sql.files.maxPartitionBytes控制单分区数据量，防止单分区数据过大引发GC。
• 动态资源分配：关闭固定spark.executor.instances配置，启用动态资源分配并设置合理上下限，让Spark根据任务负载自动调整executor数量，避免资源浪费或不足。
• 数据读取优化：避免使用select *，只读取业务必需字段；确认month为分区字段，确保分区裁剪生效，避免读取全表数据。
• 冗余参数清理：移除重复配置项，调整超时参数为合理值，避免不必要的长等待。

内容的提问来源于stack exchange，提问作者mr.data_engg