Spark 2.x中spark.akka.frameSize是否生效？Netty等效参数咨询

Great question! Let's walk through what you need to know about Spark's RPC configuration after the switch from Akka to Netty in 2.x versions:

1. Does spark.akka.frameSize still work in Spark 2.x? #

Nope, this parameter no longer takes effect in Spark 2.x and later. The Akka-based communication layer was completely replaced with Netty starting in Spark 2.0, so all Akka-specific configuration parameters (including spark.akka.frameSize) are ignored by the framework.

2. What's the Netty equivalent for spark.akka.frameSize? #

The direct replacement for controlling maximum RPC message size is the spark.rpc.message.maxSize parameter. It serves exactly the same purpose as the old Akka setting: defining the maximum size (in megabytes) of any RPC message sent between Spark nodes (driver, executors, masters, workers).

• Default value: 128 MB
• How to set it:
  • Add to your spark-defaults.conf: spark.rpc.message.maxSize 256
  • Specify when submitting a job: spark-submit --conf spark.rpc.message.maxSize=256 ...

3. Additional Recommendations #

• If you previously tuned spark.akka.frameSize to handle large messages (like shuffles with big partitions or broadcast variables), you should apply the same value to spark.rpc.message.maxSize in Spark 2.x.
• If you encounter errors like Message size exceeds max message size, this is the first parameter you should adjust to resolve the issue.
• Keep in mind that increasing this value too much can lead to higher memory usage across your cluster, so balance it with your workload's actual needs.

内容的提问来源于stack exchange，提问作者kylin