Hadoop-2.7.3作业级排除指定NodeManager的实现方法问询

Absolutely! You can totally achieve job-specific NodeManager targeting in Hadoop 2.7.3—no more global ResourceManager-level rules that affect all jobs. Here's how to split your jobs between two NodeManager groups (group1 and group2):

Step 1: Tag Your NodeManagers into Groups #

First, you need to label each NodeManager with its respective group. This tells the ResourceManager which nodes belong to which pool.

1. For each NodeManager in group1, edit its yarn-site.xml and add:

   <property>
       <name>yarn.nodemanager.node-labels</name>
       <value>group1</value>
   </property>
   

2. For each NodeManager in group2, edit its yarn-site.xml and add:

   <property>
       <name>yarn.nodemanager.node-labels</name>
       <value>group2</value>
   </property>
   

3. Restart all updated NodeManagers to apply the labels.
4. Make sure node labels are enabled on the ResourceManager:
   Edit the RM's yarn-site.xml with these settings, then restart the RM:

   <property>
       <name>yarn.node-labels.enabled</name>
       <value>true</value>
   </property>
   <property>
       <name>yarn.node-labels.manager-class</name>
       <value>org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager</value>
   </property>
   

Step 2: Assign Jobs to Specific Groups #

Now you can use a job-level property to tell each job exactly which NodeManager group to run on. There are two easy ways to do this:

Option 1: Command-Line Submission #

When launching your job, pass the label expression as a system property:

• For job1 (targeting group1):

  hadoop jar your-job-jar-file.jar YourMainClass -D mapreduce.job.node-label-expression=group1
  

• For job2 (targeting group2):

  hadoop jar your-job-jar-file.jar YourMainClass -D mapreduce.job.node-label-expression=group2
  

Option 2: Hardcode in Job Code #

If you want to bake the group target directly into your job's driver code, set the property on the Job configuration:

// For job1 targeting group1
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "Job1");
job.getConfiguration().set("mapreduce.job.node-label-expression", "group1");

// ... rest of your job setup (mapper, reducer, input/output paths, etc.)

Repeat the same logic for job2, replacing group1 with group2.

Key Notes #

• The mapreduce.job.node-label-expression property is job-specific—it only affects the job you set it on, not other jobs running on the cluster.
• Hadoop 2.7.3 fully supports node labels, so this approach is guaranteed to work with your version.
• This setting overrides any global node label configurations on the ResourceManager, so you don't have to worry about conflicting global rules.

内容的提问来源于stack exchange，提问作者Rahul