基于Ambari集群配置Spark动态分配参数的取值范围测算

Hey there! Let's work through the reasonable Spark dynamic allocation parameter ranges for your Ambari cluster, based on your specs: 21 Worker nodes (32GB RAM, 16 vCPUs each), plus 3 Masters and 3 Kafka nodes. We'll focus on the Worker nodes since that's where Spark executors run.

First, we need to set aside resources for the operating system, YARN NodeManager, and other minor background processes on each Worker:

• Memory: Reserve 3-4GB for OS + 1GB for YARN NodeManager → ~27-28GB of RAM available for Spark per Worker.
• CPU: Reserve 2 vCPUs for system processes → 14 vCPUs available for Spark per Worker.

We’ll cover two typical setups based on task type—pick the one that matches your workload, or adjust for mixed loads.

Scenario 1: CPU-Intensive Tasks #

Optimize for tasks that use heavy computation (e.g., joins, aggregations):

• spark.executor.cores: 4 (balances parallelism and context-switching overhead)
• Per-Worker executor count: 3 (14 available vCPUs ÷ 4 ≈ 3)
• spark.executor.memory: 8-10GB (we’ll use 9GB as a sweet spot: 28GB available RAM ÷ 3 ≈ 9GB)
• spark.executor.memoryOverhead: 1-2GB (reserve for off-heap memory; 1GB is sufficient here, ~10% of executor memory)

Scenario 2: IO-Intensive Tasks #

Optimize for tasks with heavy disk/network IO (e.g., reading large datasets, writing to storage):

• spark.executor.cores: 2 (fewer cores per executor means more executors can run concurrently during IO waits)
• Per-Worker executor count: 7 (14 available vCPUs ÷ 2 = 7)
• spark.executor.memory: 3-5GB (4GB is ideal: 28GB available RAM ÷ 7 = 4GB)
• spark.executor.memoryOverhead: 512MB-1GB (1GB is safer to avoid off-heap errors)

First, ensure spark.dynamicAllocation.enabled=true (this is usually configurable in Ambari’s Spark service settings). Then set these ranges based on the scenarios above:

For CPU-Intensive Setup (Total Cluster Executors: 21 Workers × 3 = 63) #

• spark.dynamicAllocation.minExecutors: 6-13 (10%-20% of total available executors—keeps a baseline for small, frequent tasks)
• spark.dynamicAllocation.maxExecutors: 50-57 (80%-90% of total available executors—leaves buffer for other YARN jobs)
• spark.dynamicAllocation.initialExecutors: 10-20 (starts with a middle ground between min and max for typical workloads)

For IO-Intensive Setup (Total Cluster Executors: 21 Workers × 7 = 147) #

• spark.dynamicAllocation.minExecutors: 15-29 (10%-20% of total available executors)
• spark.dynamicAllocation.maxExecutors: 118-132 (80%-90% of total available executors)
• spark.dynamicAllocation.initialExecutors: 20-30 (balances startup speed and resource efficiency)

• spark.shuffle.service.enabled=true: Mandatory for dynamic allocation—this keeps shuffle data intact when executors are destroyed, avoiding redundant computation. Ambari usually enables this by default.
• spark.dynamicAllocation.executorIdleTimeout: 60-300 seconds (default is 60s; set to 300s if your tasks have longer gaps between runs to avoid frequent executor teardown/creation)
• spark.dynamicAllocation.cachedExecutorIdleTimeout: 3600 seconds (1 hour—keep executors with cached data around longer, since reloading cached data is costly)

• If you have mixed workloads, split the difference: e.g., 3 cores per executor, 4 executors per Worker (12 vCPUs used), 7GB executor memory, 1GB overhead.
• Monitor YARN’s resource usage via Ambari’s metrics dashboard. If jobs regularly hit the maxExecutors limit and other jobs are waiting, you can increase max to 95% of total available executors (e.g., 60 for CPU setup, 140 for IO setup)
• Align these parameters with your YARN queue quotas—if Spark is restricted to a specific queue in Ambari, ensure the max executors don’t exceed the queue’s allocated resources.

内容的提问来源于stack exchange，提问作者shalom