MasterNode在Spark集群中的定位及YARN模式下Spark配置问询

Hey there! Let's break down your questions one by one based on your 1 Master + 2 Core Node cluster setup with YARN as the resource manager.

1. Should the Master Node be included when calculating Executor count? #

Short answer: No.

In a YARN-managed cluster, your Master Node runs the YARN ResourceManager (RM)—this is the "brain" of the cluster, responsible for scheduling resources, managing applications, and coordinating NodeManagers on Core Nodes. Running Spark executors on the Master Node would compete for resources with the RM, which could lead to instability or even downtime for the entire cluster.

You should only allocate executor resources to your Core Nodes, as those are dedicated for compute workloads.

2. Do I need to reserve 1 core per node for YARN management? #

Absolutely—this is a critical best practice.

Every node (including Core Nodes) runs the YARN NodeManager (NM), which handles container lifecycle, resource reporting, and other local management tasks. If you consume all available CPU cores on a node for Spark executors, the NodeManager might not have enough resources to operate properly, leading to delayed container scheduling or node unavailability.

Here's how to implement this:

• In YARN's yarn-site.xml, set yarn.nodemanager.resource.cpu-vcores to the total number of physical cores on the node minus 1 (e.g., if a Core Node has 8 cores, set this to 7).
• Alternatively, when configuring Spark, calculate spark.executor.cores such that the sum of executor cores per node plus 1 (for NM) doesn't exceed the total cores on the node.

3. Are there special Spark configurations needed for the Master Node? #

Not really—but there are a few key considerations:

• Don't run Spark workers on the Master Node: Since you're using YARN, Spark doesn't run its own worker processes anyway (YARN manages containers for executors). Even if you had standalone Spark mode enabled, avoid configuring the Master Node as a worker to protect the RM's resources.
• Driver placement: If running Spark in client mode, avoid running the driver on the Master Node if it's resource-intensive (e.g., large data shuffles or heavy UI overhead). For cluster mode, YARN will automatically schedule the driver on a Core Node, which is ideal.
• Point Spark to the YARN ResourceManager: Ensure your Spark config (spark-defaults.conf) has spark.yarn.resourcemanager.address set to your Master Node's RM address (e.g., master-node:8032). This is how Spark communicates with YARN to request resources.

Quick Example for Your Cluster #

Suppose each Core Node has 4 physical cores:

• Reserve 1 core per node for NodeManager → 3 cores available per Core Node.
• Set spark.executor.cores to 3 → you'll get 1 executor per Core Node, totaling 2 executors.
• This ensures efficient resource usage without starving YARN's management services.

内容的提问来源于stack exchange，提问作者simplycoding