如何获取Hadoop Yarn已完成作业的调度器并验证调度器配置

Hey there! I’ve worked with Hadoop YARN schedulers quite a bit, so let’s break down your questions with practical, actionable steps:

1. How to Get the Scheduler Used by Completed Hadoop YARN Jobs #

Completed jobs aren’t accessible via standard active YARN CLI commands, but you have a few reliable ways to retrieve their scheduler info:

• Check the Job History Server UI
  Access the Job History Server (default port 19888) in your browser. Navigate to the completed job’s detail page—look under sections like Job Configuration or Scheduler Info for entries like yarn.resourcemanager.scheduler.class or explicit mentions of the scheduler type (e.g., FIFO).

• Parse Job Logs
  Fetch the full logs for your completed job with this command, then search for scheduler-related keywords:

  yarn logs -applicationId <your-application-id>
  

  Look for lines like INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler: Assigned application app_xxxx to queue default—this directly tells you which scheduler handled the job.

• Use MapReduce Job Status Command
  For MapReduce-specific jobs, this command pulls metadata (including scheduler details) even after completion:

  mapred job -status <your-job-id>
  

  Scan the output for fields linked to the queue and scheduler.

2. Verify FIFO Scheduler Configuration Changes via Job ID #

To confirm your mapred-site.xml and scheduler config edits are working, and check which scheduler a job/queue uses, try these methods:

• Check Running/Recent Jobs with YARN CLI
  For jobs that are still running or recently finished (before being purged from YARN’s active list), run:

  yarn application -status <your-application-id>
  

  In the output, look for sections like:

  Queue: default
  Scheduler Info: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler
  This confirms the job is using the FIFO scheduler.

• Inspect Queue Configuration
  Verify your queue’s scheduler settings with:

  yarn queue -status <your-queue-name>
  

  Look for entries referencing the scheduler class—for FIFO, it should point to the default FIFO scheduler class (or your customized version).

• Validate via ResourceManager UI
  Open the ResourceManager UI (default port 8088), go to the Cluster > Scheduler tab. Here you’ll see the scheduler type for each queue—if your FIFO changes are active, the queue will explicitly show as using the FIFO Scheduler. Cross-check this with your job’s assigned queue.

• Submit a Test Job
  The most concrete validation is to run a small test job (like a simple WordCount) and immediately run yarn application -status on its ID. This will show you the exact scheduler handling the job, confirming your config changes took effect.

内容的提问来源于stack exchange，提问作者mani