关于Hadoop 2.9.0中YARN提交不同优先级作业的问题咨询

Hey there! Let's dig into why your YARN application priorities are stuck at 0, even though you've tried setting them via YarnClient and the CLI. First, rest assured that Hadoop 2.9.0 does support application priorities (as you saw in JIRA tickets), so the issue is likely with configuration, implementation, or a misunderstanding of how priorities work in YARN.

Common Issues & Fixes #

1. Your YARN Scheduler Isn't Configured to Support Priorities #

By default, some schedulers don't enable priority handling out of the box. Here's what to check based on your scheduler:

For CapacityScheduler:

Add or verify these properties in yarn-site.xml:

<property>
  <name>yarn.scheduler.capacity.enable-priority</name>
  <value>true</value>
</property>
<property>
  <name>yarn.scheduler.capacity.application-priority.update.enable</name>
  <value>true</value>
</property>
<property>
  <name>yarn.scheduler.capacity.maximum-priority</name>
  <value>10</value> <!-- Adjust as needed; default is 10 -->
</property>

For FairScheduler:

Add or verify these in yarn-site.xml:

<property>
  <name>yarn.scheduler.fair.enable-priority</name>
  <value>true</value>
</property>
<property>
  <name>yarn.scheduler.fair.max-priority</name>
  <value>10</value>
</property>

If you're using FIFO Scheduler, it does support priorities but requires no additional config—though FIFO will only prioritize higher-priority jobs when resources are available.

2. You're Setting Priorities Outside the Allowed Range #

YARN's ApplicationPriority has a default valid range of 0 (lowest) to 10 (highest). If you set a priority outside this range, it will be clamped to 0 (if too low) or the maximum allowed value (if too high). Double-check that you're setting a value within the range defined by your scheduler's maximum-priority config.

3. Incorrect Implementation with YarnClient #

Make sure you're setting the priority correctly in your ApplicationSubmissionContext before submitting the application. Here's a correct example:

// Create priority instance with desired value (e.g., 5)
Priority priority = Priority.newInstance(5);

// Attach to ApplicationSubmissionContext
ApplicationSubmissionContext appSubmissionContext = yarnClient.createApplicationSubmissionContext();
appSubmissionContext.setPriority(priority);
appSubmissionContext.setQueue("your-queue-name"); // Ensure queue supports priorities

// Submit the application
yarnClient.submitApplication(appSubmissionContext);

A common mistake is setting the priority after submitting the application—once submitted, you need to use the CLI update command instead.

4. CLI Update Command Issues #

When using updateApplicationPriority, ensure:

• You're using the correct command syntax:

  yarn application -updatePriority <priority-value> <application-id>
  

• The application is in a state that allows priority updates (e.g., ACCEPTED or RUNNING—completed or failed apps can't be modified).
• Your user account has the necessary permissions to modify application priorities. YARN's access control lists (ACLs) might block this action if you don't have admin or modify rights for the queue or application.

Clarifying ApplicationPriority Concepts #

It's easy to misinterpret how YARN priorities work—here's a quick reality check:

• Priorities are relative, not absolute: A higher priority job won't immediately steal resources from lower-priority jobs unless your scheduler is configured for preemption (e.g., FairScheduler's preemption or CapacityScheduler's elastic queues).
• Queue rules take precedence: If your queue has a minimum capacity guarantee, lower-priority jobs in that queue will still get resources before higher-priority jobs in a different queue that's already using its allocated capacity.
• Priority affects scheduling order: When resources become available, the scheduler will prioritize higher-priority jobs for resource allocation first—but only within the constraints of queue capacities and scheduler policies.

Final Steps to Debug #

1. Restart the ResourceManager after updating any yarn-site.xml configs.
2. Check the ResourceManager logs for errors related to priority handling (look for keywords like priority or ApplicationPriority).
3. Verify the priority of your application using:

   yarn application -status <application-id>
   

   This will show the current priority value—if it's still 0, your config or implementation is missing something.

内容的提问来源于stack exchange，提问作者jm.