如何确定Ambari 2.6集群中yarn.scheduler.maximum-allocation-vcores的取值

Great question! Let's break down how to pick the optimal value for yarn.scheduler.maximum-allocation-vcores based on your cluster setup (3 Worker nodes, 16 physical cores each, with yarn.nodemanager.resource.cpu-vcores already set to 16 per node).

First, let's clarify what this parameter does: it defines the maximum number of vcores that YARN's scheduler can allocate to a single Container. This value directly balances two key goals: meeting the resource needs of your jobs, and keeping your cluster's CPU resources utilized efficiently.

Here's how to narrow down the right value:

1. Align with your workload type #

This is the most critical factor:

• Batch processing (Spark, MapReduce, etc.): If your jobs are mostly large, long-running tasks that benefit from more CPU per container (e.g., Spark Executors with 8-12 cores), set this value to match that typical need. A safe starting point is 8-12 (50%-75% of your node's total vcores). Avoid setting it to the full 16 unless you intentionally want single containers to consume an entire node's CPU—this can starve smaller jobs and hurt overall cluster utilization.
• Interactive/small jobs (Hive ad-hoc queries, small data pipelines): If most jobs are lightweight and only need 2-4 cores to finish quickly, set this lower (e.g., 4). This lets your nodes run more containers in parallel, reducing job queue times and keeping CPU utilization high.

2. Consider total cluster capacity and concurrency needs #

Your cluster has a total of 3 * 16 = 48 available vcores.

• If you prioritize high concurrency (running as many jobs as possible), go with a smaller value (e.g., 4). This theoretically allows up to 12 concurrent containers (real-world numbers will be slightly lower due to other resource constraints like memory).
• If you prioritize fast execution of large jobs, opt for a larger value (e.g., 12). This lets big jobs grab more CPU to finish quicker, but limits the number of concurrent jobs to around 4.

3. Match your job's typical resource requests #

Look at the resource settings your jobs already use:

• For MapReduce, check mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores
• For Spark, check spark.executor.cores

Set yarn.scheduler.maximum-allocation-vcores to be at least as large as the highest value used by your jobs. This ensures your jobs don't get rejected because their CPU request exceeds the scheduler's maximum allocation. A common practice is to set it to 10-20% higher than your typical job's CPU request, to leave room for occasional larger jobs.

4. Test and adjust incrementally #

There's no one-size-fits-all value—start with a middle ground (like 8) and monitor your cluster:

• Check if jobs are getting the CPU resources they need (no "resource request exceeded" errors)
• Track overall CPU utilization across nodes (avoid idle cores or constant 100% usage)
• Watch job queue times (if small jobs are waiting too long, lower the value; if big jobs are slow, raise it)

Tweak the value based on what you observe until you hit the sweet spot for your specific workload.

内容的提问来源于stack exchange，提问作者shalom