关于Hadoop中mapreduce.framework.name参数及版本架构的技术咨询

Hey there! Your understanding of Hadoop 1 and Hadoop 2 (YARN) execution environments is completely correct—nice work grasping those core architectural differences! Let me break this down a bit more clearly and dive into the mapreduce.framework.name parameter you asked about.

Hadoop Version Daemon Architecture Breakdown #

First, let’s confirm your initial understanding with a structured breakdown:

• Hadoop 1.x: The MapReduce execution layer depends entirely on two daemons:
  • JobTracker: The central controller for all cluster jobs—handles scheduling tasks to worker nodes, tracking job progress, and managing task failures.
  • TaskTracker: Runs on every worker node, executes the Map/Reduce tasks assigned by the JobTracker, and sends status updates back to the central JobTracker.
• Hadoop 2.x (YARN): Redesigned the cluster resource management to be more flexible and scalable, using these key daemons:
  • ResourceManager: The global cluster resource overseer—allocates CPU, memory, and other resources to applications, and manages overall cluster resource scheduling.
  • NodeManager: Runs on each worker node, manages the node's local resources, launches task containers, and reports node health and resource usage to the ResourceManager.
  • ApplicationMaster: A per-application daemon (one for every job or application submitted) that negotiates resources with the ResourceManager, works with NodeManagers to launch tasks, and monitors the application's lifecycle.

Deep Dive into the mapreduce.framework.name Configuration Parameter #

This parameter is the switch that tells Hadoop which execution framework to use for MapReduce jobs, and it’s directly linked to the architectures we just covered.

Key Values & Use Cases #

• local: Runs MapReduce jobs entirely on your local machine, no cluster daemons required. Perfect for testing small jobs or debugging without spinning up a full cluster.
• classic: Enables the legacy Hadoop 1.x JobTracker/TaskTracker architecture. You’d only use this if you’re maintaining an older Hadoop 1.x cluster.
• yarn: Enables the YARN framework (Hadoop 2.x+). This is the standard setting for modern Hadoop clusters, as it unlocks YARN’s benefits like multi-application resource sharing, better resource isolation, and support for non-MapReduce workloads (like Spark or Flink) on the same cluster.

Configuration Location #

You’ll set this parameter in the mapred-site.xml file (located in your Hadoop configuration directory, usually $HADOOP_CONF_DIR) using this XML property block:

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

Why It Matters #

Changing this value completely alters how your MapReduce jobs run. For example, switching from classic to yarn lets you leverage YARN’s distributed resource management, which is far more scalable and flexible than the old single JobTracker model (which was a major bottleneck in Hadoop 1.x).

内容的提问来源于stack exchange，提问作者CuriousMind