AWS环境Hadoop 2.6.5上传HDFS提示0个DataNode运行问题

在AWS环境的Hadoop 2.6.5集群中，执行hdfs dfs -put A.txt /input/上传文件时出现以下错误：

23/11/08 22:19:25 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /input/A.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1559)
        ...
put: File /input/A.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.

关键现象 #

• Master节点进程（jps输出）：NameNode、SecondaryNameNode、ResourceManager正常运行

ubuntu@ip-172-31-17-143:~$ jps
1155 SecondaryNameNode
2044 Jps
957 NameNode
1278 ResourceManager

• Slave节点进程（jps输出）：DataNode、NodeManager正常运行

ubuntu@ip-172-31-23-143:~$ jps
1504 DataNode
1625 NodeManager
1786 Jps

• hdfs dfsadmin -report显示集群无可用容量，无DataNode：

ubuntu@ip-172-31-17-143:~$ hdfs dfsadmin -report
Configured Capacity: 0 (0 B)
Present Capacity: 0 (0 B)
DFS Remaining: 0 (0 B)
DFS Used: 0 (0 B)
DFS Used%: NaN%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

• NameNode日志确认无DataNode注册：

2023-11-11 23:37:50,542 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 0 racks and 0 datanodes
...
2023-11-11 23:40:52,839 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 172.31.46.192:51584 Call#5 Retry#0
java.io.IOException: File /input/A.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.
    ...

相关配置文件 #

core-site.xml #

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
...
<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://master:9000/</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>file:/home/ubuntu/hadoop-2.6.5/tmp</value>
        </property>
</configuration>

hdfs-site.xml #

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
...
<configuration>
<property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/home/ubuntu/hadoop-2.6.5/dfs/name</value>
</property>
<property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/home/ubuntu/hadoop-2.6.5/dfs/data</value>
</property>
<property>
        <name>dfs.replication</name>
        <value>3</value>
</property>
</configuration>

yarn-site.xml #

<?xml version="1.0"?>
...
<configuration>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
                <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>
        <property>
                <name>yarn.resourcemanager.address</name>
                <value>master:8032</value>
        <property>
                <name>yarn.resourcemanager.scheduler.address</name>
                <value>master:8030</value>
        </property>
        ...
</configuration>

hadoop-env.sh #

# Licensed to the Apache Software Foundation (ASF) under one
...
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_PREFIX=/home/ubuntu/hadoop-2.6.5
...
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
...

原因分析及解决方法 #

1. 主机名解析问题 #

• 原因：Slave节点无法正确解析Master节点的master主机名，或Master节点无法识别Slave节点的主机名/IP。
• 解决步骤：
  • 在所有节点的/etc/hosts文件中添加Master和Slave节点的IP与主机名映射，例如：

    172.31.17.143 master
    172.31.23.143 slave1
    

  • 验证解析：在Slave节点执行ping master，在Master节点执行ping slave1，确认能正常连通。

2. DataNode与NameNode的集群ID不匹配 #

• 原因：NameNode格式化后，Slave节点的DataNode保留了旧的集群ID，导致无法注册到新的NameNode。
• 解决步骤：
  • 停止所有Hadoop服务：stop-all.sh
  • 删除Slave节点的DataNode数据目录：rm -rf /home/ubuntu/hadoop-2.6.5/dfs/data
  • 删除所有节点的临时目录：rm -rf /home/ubuntu/hadoop-2.6.5/tmp
  • 重新格式化NameNode：hdfs namenode -format
  • 重启Hadoop服务：start-all.sh

3. AWS安全组或防火墙拦截端口 #

• 原因：Hadoop通信所需端口未开放，导致DataNode无法向NameNode注册。
• 解决步骤：
  • 在AWS安全组中开放以下端口：
    • NameNode端口：9000、50070
    • DataNode端口：50010、50020、50075
    • YARN相关端口：8032、8030、8088等
  • 关闭节点防火墙（或添加规则）：sudo ufw disable（Ubuntu系统）

4. 目录权限问题 #

• 原因：DataNode的数据目录或Hadoop临时目录权限不足，导致DataNode无法写入数据，注册失败。
• 解决步骤：
  • 修改目录权限为Hadoop运行用户（此处为ubuntu）：

    sudo chown -R ubuntu:ubuntu /home/ubuntu/hadoop-2.6.5/dfs/data
    sudo chown -R ubuntu:ubuntu /home/ubuntu/hadoop-2.6.5/tmp
    

  • 重启DataNode服务：hadoop-daemon.sh stop datanode && hadoop-daemon.sh start datanode

5. DataNode配置错误 #

• 原因：hdfs-site.xml或core-site.xml配置有误，导致DataNode无法连接NameNode。
• 解决步骤：
  • 确认所有节点的core-site.xml中fs.defaultFS配置一致，均指向hdfs://master:9000/
  • 确认hdfs-site.xml中dfs.datanode.data.dir路径存在且正确
  • 重启DataNode服务

内容的提问来源于stack exchange，提问作者Rahul