Hadoop 2.7.5 AM Container启动错误求助

Hey there, let’s dig into this AM Container startup issue you’re facing with Hadoop 2.7.5 on MacOS High Sierra. Your jps output confirms all core Hadoop daemons are running, so the problem is likely tied to configuration mismatches, resource limits, or environment inconsistencies specific to MacOS. Let’s break down the possible fixes step by step:

1. Ensure Java Version & JAVA_HOME Consistency #

Hadoop 2.7.5 only supports Java 7 or 8—newer versions (9+) will cause compatibility issues with the ApplicationMaster.

• Run java -version to confirm your active Java version. If it’s 9 or higher, switch to Java 8 using tools like jenv or update your system’s default Java path.
• Verify JAVA_HOME is set correctly in both your shell environment and Hadoop’s config:
  • Check your shell: echo $JAVA_HOME
  • Open $HADOOP_HOME/etc/hadoop/hadoop-env.sh and ensure the export JAVA_HOME= line points to the correct Java 8 directory (e.g., /Library/Java/JavaVirtualMachines/jdk1.8.0_291.jdk/Contents/Home).

2. Adjust YARN Resource Allocation Settings #

AM Container failures often stem from insufficient memory or strict virtual memory checks. Update your $HADOOP_HOME/etc/hadoop/yarn-site.xml with these properties:

<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>2048</value> <!-- Minimum 2GB for basic jobs -->
</property>
<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>512</value>
</property>
<property>
  <name>yarn.nodemanager.vmem-check-enabled</name>
  <value>false</value> <!-- Temporarily disable for testing -->
</property>

After saving, restart YARN services:

stop-yarn.sh && start-yarn.sh

3. Inspect ApplicationMaster Logs for Exact Errors #

The most useful info is in the AM’s stderr logs. Navigate to the user logs directory:

cd $HADOOP_HOME/logs/userlogs

Look for the latest directory named after your Pi job (it’ll start with application_), then open the stderr file. Common red flags here include:

• "Container killed due to memory limits" (adjust YARN memory settings above)
• "Permission denied" (fix HDFS permissions next)

4. Fix HDFS Permissions #

Incorrect HDFS permissions can prevent the AM from accessing necessary directories. Run these commands to set proper permissions:

hadoop fs -chmod 755 /
hadoop fs -chmod 777 /tmp
hadoop fs -chown -R $(whoami):$(whoami) /user/$(whoami)

5. Disable IPv6 (MacOS-Specific Fix) #

MacOS High Sierra’s default IPv6 setup can cause networking glitches with Hadoop. Add these lines to $HADOOP_HOME/etc/hadoop/hadoop-env.sh:

export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true"
export YARN_OPTS="-Djava.net.preferIPv4Stack=true"

Restart all Hadoop services after this change:

stop-all.sh && start-all.sh

If none of these steps resolve the issue, share the full error output from your Pi job command and the content of the AM’s stderr log—this will help pinpoint the exact root cause.

内容的提问来源于stack exchange，提问作者AbdealiLoKo