Dataproc 1.4集群Spark外部Shuffle服务连接失败原因排查及优化方案咨询

Let's break down your problem and evaluate your proposed solutions, plus add some additional optimizations that might get to the root of the issue faster.

The core issue here is clear from your supplementary error message: your External Shuffle Service (ESS) on port 7337 is either overloaded, or the network timeout is too short for the backlog of shuffle requests. The connection being closed after 120 seconds of inactivity (while requests are still outstanding) confirms that ESS can't keep up with the load from your 30000 partitions across 100 nodes.

Evaluating Your Proposed Solutions #

1. Increase Cluster Node Count #

This is absolutely a valid approach. Adding more nodes will distribute the shuffle load across more ESS instances, reducing the number of shuffle partitions each node's ESS has to handle (right now that's ~300 partitions per node). Less per-node load means fewer bottlenecks, faster request processing, and fewer dropped connections. Just make sure you scale proportionally to your data volume—don't add nodes unnecessarily, but matching node count to shuffle workload will directly alleviate ESS pressure.

2. Disable External Shuffle Service #

This works too, but only if your use case allows it. ESS exists primarily to preserve shuffle files when executors are terminated (like during preemption or autoscaling). Since you're not using preemptible VMs and have autoscaling disabled, your executors should stay running for the full duration of the job. Disabling ESS will store shuffle files locally on executors instead of relying on the dedicated service, removing that bottleneck entirely.

Caveat: If your job has stage retries, or if executors crash unexpectedly (even without scaling), you'll lose shuffle files and hit failures. Only use this if your job is stable and executors don't drop out mid-run.

3. Increase spark.network.timeout #

This is a temporary workaround, not a fix. Extending the timeout gives ESS more time to process backlogged requests, but it doesn't address the underlying overload. You might delay the failure, but you'll likely still hit connection issues once the new timeout window is reached, and your job runtime will increase as tasks wait longer for shuffle data. Use this only as a short-term band-aid while you implement more permanent fixes.

Additional Optimizations to Fix the Root Cause #

Address Data Skew #

A common culprit behind ESS overload is data skew—if a small number of shuffle partitions are significantly larger than others, they'll overwhelm the ESS instance handling them. Check your job's shuffle metrics to identify skewed keys, then fix it with:

• Salting: Add a random suffix to skewed keys to split them into smaller partitions
• Filtering: Remove or aggregate overly large keys before shuffling
• Custom Partitioning: Use a custom partitioner to distribute skewed data more evenly

Tune ESS-Specific Configurations #

Adjust these Spark settings to boost ESS performance:

• spark.shuffle.service.threads: Increase the number of threads ESS uses to handle shuffle requests (default is 4; try 8-16 for high-load jobs)
• spark.shuffle.service.index.cache.size: Expand the cache for shuffle index files to reduce disk I/O (default is 100; increase to 500 or 1000 if you have enough memory)
• spark.shuffle.io.maxRetries & spark.shuffle.io.retryWait: Increase retry counts and wait time for shuffle requests (e.g., set retries to 5 and wait to 5s) to handle temporary overload spikes
• spark.shuffle.service.file.buffer: Increase the buffer size for shuffle file transfers to reduce I/O overhead

Check Cluster Resource Utilization #

Verify if the nodes running ESS are hitting resource limits:

• CPU/Disk I/O: Use cluster monitoring tools to check if ESS nodes are maxing out CPU or disk throughput. If disk I/O is the bottleneck, switch to SSD storage for shuffle files.
• Memory: Ensure ESS has enough memory allocated (via spark.shuffle.service.memory.max) to cache shuffle indices and handle requests without swapping to disk.

Re-evaluate Partition Count #

While you said you've minimized partitions, 30000 across 100 nodes is 300 partitions per node. Aim for shuffle partitions that are ~128MB-256MB in size—if your partitions are much smaller than that, you're creating unnecessary overhead for ESS. If they're larger, split them further to distribute load more evenly.

Recommended Priority Order #

1. First, check for and fix data skew—this is the most impactful fix for overload issues.
2. Tune ESS-specific configurations to squeeze more performance out of your existing cluster.
3. If skew is fixed and configs are tuned but you still see overload, add more nodes to distribute the load.
4. Disable ESS only as a last resort, if your job's stability guarantees it.
5. Use spark.network.timeout only as a temporary stopgap.

内容的提问来源于stack exchange，提问作者Yann Moisan