Hey there, let's work through this container launch failure together—that generic exit code 1 can be tricky, but we can narrow it down with a few targeted checks based on your setup.

First, let's recap your scenario: you've got a 1 NameNode + 3 DataNode Hadoop cluster, and your MapReduce job is failing with a container exit code 1, with no extra details in DataNode logs. Here are the key areas to investigate:

1. Verify Local Directory Permissions & Disk Space #

Your YARN config specifies local and log directories at /usr/local/hadoop_work/yarn/local and /usr/local/hadoop_work/yarn/log, plus HDFS data directories at /usr/local/hadoop_work/hdfs/datanode.

• Run ls -ld /usr/local/hadoop_work/yarn /usr/local/hadoop_work/hdfs on all DataNodes to confirm the directories are owned by the user running Hadoop (e.g., hadoop) with permissions set to 755 (avoid 777, which can trigger security restrictions).
• Check disk space with df -h on each node—full disks will block container creation or log writing entirely.

2. Pull Aggregated YARN Logs #

You’ve enabled log aggregation (yarn.log-aggregation-enable=true), which is perfect! Use this command to fetch detailed container logs directly from HDFS (replace <APP_ID> with your application ID, which looks like application_1527096061793_0001 from your error trace):

yarn logs -applicationId <APP_ID>

These logs will include specific errors from the failed container (like missing dependencies, permission gaps, or configuration mismatches) that won’t show up in DataNode system logs.

3. Fix Incomplete MapReduce Configuration #

Your provided mapred-site.xml is truncated—it cuts off at <property> <name>mapred.chil. This is a critical issue! Incomplete config files can break container startup entirely.

• Restore the full mapred-site.xml configuration. At minimum, ensure you have these memory-related properties set (adjust values based on your node resources):

  <property>
      <name>mapreduce.map.memory.mb</name>
      <value>2048</value>
  </property>
  <property>
      <name>mapreduce.reduce.memory.mb</name>
      <value>4096</value>
  </property>
  <property>
      <name>mapred.child.java.opts</name>
      <value>-Xmx1536m</value>
  </property>
  

• Double-check that all properties are properly closed with </property> and the file ends with </configuration>.

4. Validate Hostname Resolution & SSH Access #

Container launch relies on seamless communication between nodes:

• On every DataNode, run ping NameNode to confirm the hostname resolves to the correct IP address. If not, add an entry for NameNode in /etc/hosts on all nodes.
• Test passwordless SSH from each DataNode to the NameNode (and vice versa) with ssh NameNode—if prompted for a password, fix your SSH key setup for the Hadoop user.

5. Check HDFS Staging Directory Permissions #

Your mapred-site.xml sets yarn.app.mapreduce.am.staging-dir to /user/app. Ensure this directory exists and has proper permissions:

# Create the directory if missing
hdfs dfs -mkdir -p /user/app
# Set ownership to your Hadoop user
hdfs dfs -chown <hadoop-user>:<hadoop-group> /user/app
# Verify permissions
hdfs dfs -ls /user

Without write access here, the Application Master can’t stage job resources, leading to container failures.

6. Inspect Container Executor Permissions #

If you’re using DefaultContainerExecutor, the local YARN directories must not have overly permissive permissions. Stick to 755 for directories and 644 for files, owned by the user running YARN (usually yarn or your Hadoop user)—777 permissions can trigger security blocks.

Start with these steps—most exit code 1 container failures boil down to permission issues, incomplete configs, or resource constraints. Let me know if you find anything specific in the aggregated logs!

内容的提问来源于stack exchange，提问作者Sif