Spark集群模式下YARN的数据本地性感知与Executor精准分配机制

Let’s break down your questions clearly—this gets right to the core of how Spark and YARN optimize for data locality:

1. How does YARN precisely allocate an Executor on NODE 1 (where the HDFS block resides)? #

The magic happens through tight coordination between Spark’s job orchestration and YARN’s resource scheduling:

• When you submit your Spark job to YARN in cluster mode, the Spark client first sends the job package to the YARN ResourceManager (RM).
• The RM spins up a Spark ApplicationMaster (AM) on a cluster node—this AM becomes the "boss" of your job’s lifecycle.
• First, the AM figures out where your records.txt blocks live (more on that in the next section). It learns one replica of the 128MB block is on NODE 1.
• To avoid slow cross-node data transfers, the AM sends a resource request to the YARN RM specifically asking for a Container (which hosts the Executor) on NODE 1. It explicitly marks NODE 1 as the preferred node in the request.
• The YARN RM checks in with NODE 1’s NodeManager (NM) to see if there’s enough free memory/CPU to run the Executor. If yes, the RM tells NODE 1’s NM to launch the Container and start the Executor inside it.
• Once the Executor is up and running, Spark schedules the data processing task directly on this Executor, so it reads the records.txt block locally from NODE 1’s HDFS DataNode.

2. Who tells YARN that a block of records.txt is on NODE 1? #

YARN doesn’t fetch this info directly—instead, the Spark ApplicationMaster gets the block location data from the HDFS NameNode:

• The NameNode is HDFS’s "metadata manager"—it keeps a master map of every file’s blocks and which DataNodes (like NODE 1) store each replica.
• When the Spark AM needs to plan task execution, it sends a query to the NameNode asking where the blocks of records.txt are stored. The NameNode responds with the list of DataNodes hosting each block replica.
• The AM then uses this location data to guide YARN: it tells the YARN RM to prioritize allocating Containers on nodes with the data (NODE 1 here).

A quick side note: If NODE 1 didn’t have free resources, YARN might allocate the Executor on another node, and Spark would have to pull the block over the network—but it always tries the data-local node first to optimize performance.

内容的提问来源于stack exchange，提问作者Surender Raja