Glue运行PySpark报Remote RPC Client Disassociated错误求解决方案

在Glue 3.0（对应Spark 3.1）环境运行PySpark代码时，出现如下报错：

Remote RPC Client Disassociated.Likely due to containers exceeding thresholds, or network issues
（远程RPC客户端断开连接，可能是容器超出阈值或网络问题导致）

已尝试调整多种Spark RPC及内存类配置参数：

• executor内存分别试过1g、20g、30g、64g
• driver内存分别试过20g、30g、64g

但问题仍未解决，当前Spark配置如下：

spark= (SparkSession
    .builder
    .appName("jkTestApp")
    .config("spark.sql.sources.partitionOverwriteMode", "dynamic")
    .config("spark.sql.autoBroadcastJoinThreshold","-1")
    .config("spark.rpc.message.maxSize", "512")
    .config("spark.driver.memory","64g")
    .config("spark.executor.memory","1g")
    .config("spark.executor.memoryOverhead","512")
    .config("spark.rpc.numRetries","10")
    .getOrCreate()
     )

内容的提问来源于stack exchange，提问作者Jatin