能否让Hadoop不同组件使用不同Java版本（含修改版）运行？

Absolutely, this is totally feasible—and it’s a common setup when you need a specialized runtime for your MapReduce workloads while keeping core Hadoop services on a stable, standard JVM like Java 1.8. Here’s how to make it work:

Step 1: Configure Core Hadoop Services to Use Java 1.8 #

First, ensure your namenode, datanode, and scheduler (part of the ResourceManager) run on Java 1.8. You can set this globally for Hadoop core services by:

• Editing the hadoop-env.sh file (usually in $HADOOP_HOME/etc/hadoop) and setting HADOOP_JAVA_HOME=/path/to/java1.8
• Alternatively, setting the system-wide JAVA_HOME to Java 1.8 before starting Hadoop services—this will be inherited by core components by default.

Step 2: Set Up MapReduce Containers to Use Your Custom Java Build #

To force MapReduce map/reduce tasks (running in containers) to use your modified Java environment, you’ll need to adjust MapReduce configuration properties in mapred-site.xml:

• Add or update the mapreduce.map.java.home property to point to your custom Java’s installation path:

  <property>
      <name>mapreduce.map.java.home</name>
      <value>/path/to/modified/java</value>
  </property>
  

• Do the same for reduce tasks with mapreduce.reduce.java.home:

  <property>
      <name>mapreduce.reduce.java.home</name>
      <value>/path/to/modified/java</value>
  </property>
  

• If your custom Java requires additional environment variables (like custom library paths), you can pass them using mapreduce.admin.user.env:

  <property>
      <name>mapreduce.admin.user.env</name>
      <value>JAVA_HOME=/path/to/modified/java;LD_LIBRARY_PATH=/path/to/custom/libs</value>
  </property>
  

Key Notes to Keep in Mind #

• Consistent Paths: Make sure the custom Java path is identical across all worker nodes in your cluster—containers will run on these nodes, so they need access to the modified JVM.
• Compatibility Check: Verify that your modified Java is compatible with the Hadoop version you’re running. For example, if your Hadoop version targets Java 8 bytecode, ensure your custom JVM can execute that code without issues.
• Permissions: Ensure the Hadoop runtime user (usually hadoop) has read and execute permissions on the custom Java installation directory.

内容的提问来源于stack exchange，提问作者Kristianto Karim