Docker部署Hadoop集群重启后DataNode启动失败求助

问题现象 #

Docker部署Hadoop集群，首次启动运行稳定，但重启集群后DataNode仅启动几秒就停止，日志核心报错为java.io.IOException: All specified directories have failed to load，排查发现重启后DataNode的clusterid与NameNode不一致，常规方案无效。

错误日志 #

2023-08-17 10:25:48 2023-08-17 07:25:48 ERROR DataNode:903 - Initialization failed for Block pool <registering> (Datanode Uuid 11f11079-a4eb-4e8b-9023-3b14351c7e85) service to namenode/172.29.0.2:8020. Exiting. 
2023-08-17 10:25:48 java.io.IOException: All specified directories have failed to load.
2023-08-17 10:25:48     at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:562)
2023-08-17 10:25:48     at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:2059)
2023-08-17 10:25:48     at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1995)
2023-08-17 10:25:48     at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:394)
2023-08-17 10:25:48     at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:312)
2023-08-17 10:25:48     at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:891)
2023-08-17 10:25:48     at java.lang.Thread.run(Thread.java:748)
2023-08-17 10:25:48 2023-08-17 07:25:48 WARN  DataNode:931 - Ending block pool service for: Block pool <registering> (Datanode Uuid 11f11079-a4eb-4e8b-9023-3b14351c7e85) service to namenode/172.29.0.2:8020
2023-08-17 10:25:48 2023-08-17 07:25:48 INFO  DataNode:102 - Removed Block pool <registering> (Datanode Uuid 11f11079-a4eb-4e8b-9023-3b14351c7e85)
2023-08-17 10:25:50 2023-08-17 07:25:50 WARN  DataNode:3256 - Exiting Datanode
2023-08-17 10:25:50 2023-08-17 07:25:50 INFO  DataNode:51 - SHUTDOWN_MSG: 
2023-08-17 10:25:50 /************************************************************
2023-08-17 10:25:50 SHUTDOWN_MSG: Shutting down DataNode at db3b2eb8c96c/172.29.0.5
2023-08-17 10:25:50 ************************************************************

集群配置 #

docker-compose.yml #

version: "3.3.6"
services:
   namenode:
      image: apache/hadoop:3.3.6
      hostname: namenode
      command: ["hdfs", "namenode"]
      ports:
        - 9870:9870
      env_file:
        - ./config
      environment:
          ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
   datanode1:
      image: apache/hadoop:3.3.6
      command: ["hdfs", "datanode"]
      env_file:
        - ./config
   datanode2:
      image: apache/hadoop:3.3.6
      command: ["hdfs", "datanode"]
      env_file:
        - ./config  
   resourcemanager:
      image: apache/hadoop:3.3.6
      hostname: resourcemanager
      command: ["yarn", "resourcemanager"]
      ports:
         - 8088:8088
      env_file:
        - ./config
      volumes:
        - ./test.sh:/opt/test.sh
   nodemanager:
      image: apache/hadoop:3.3.6
      command: ["yarn", "nodemanager"]
      env_file:
        - ./config

config配置文件 #

CORE-SITE.XML_fs.default.name=hdfs://namenode
CORE-SITE.XML_fs.defaultFS=hdfs://namenode
HDFS-SITE.XML_dfs.namenode.rpc-address=namenode:8020
HDFS-SITE.XML_dfs.replication=1
MAPRED-SITE.XML_mapreduce.framework.name=yarn
MAPRED-SITE.XML_yarn.app.mapreduce.am.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
MAPRED-SITE.XML_mapreduce.map.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
MAPRED-SITE.XML_mapreduce.reduce.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
YARN-SITE.XML_yarn.resourcemanager.hostname=resourcemanager
YARN-SITE.XML_yarn.nodemanager.pmem-check-enabled=false
YARN-SITE.XML_yarn.nodemanager.delete.debug-delay-sec=600
YARN-SITE.XML_yarn.nodemanager.vmem-check-enabled=false
YARN-SITE.XML_yarn.nodemanager.aux-services=mapreduce_shuffle
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.maximum-applications=10000
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.maximum-am-resource-percent=0.1
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.queues=default
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.capacity=100
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.user-limit-factor=1
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.maximum-capacity=100
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.state=RUNNING
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.acl_submit_applications=*
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.acl_administer_queue=*
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.node-locality-delay=40
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.queue-mappings=
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.queue-mappings-override.enable=false

解决方案 #

根本原因 #

当前配置中NameNode和DataNode的存储目录均未持久化到宿主机，容器重启后：

• NameNode的/tmp/hadoop-root/dfs/name目录被清空，触发重新初始化并生成新的clusterid
• DataNode若残留旧存储数据，会因与NameNode的clusterid不匹配导致初始化失败

修复步骤 #

1. 修改docker-compose.yml，添加存储卷持久化
   更新后的配置给NameNode和每个DataNode挂载宿主机目录，避免容器重启后数据丢失：

   version: "3.3.6"
   services:
      namenode:
         image: apache/hadoop:3.3.6
         hostname: namenode
         command: ["hdfs", "namenode"]
         ports:
           - 9870:9870
         env_file:
           - ./config
         environment:
             ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
         volumes:
           - ./namenode-data:/tmp/hadoop-root/dfs/name
      datanode1:
         image: apache/hadoop:3.3.6
         command: ["hdfs", "datanode"]
         env_file:
           - ./config
         volumes:
           - ./datanode1-data:/tmp/hadoop-root/dfs/data
      datanode2:
         image: apache/hadoop:3.3.6
         command: ["hdfs", "datanode"]
         env_file:
           - ./config  
         volumes:
           - ./datanode2-data:/tmp/hadoop-root/dfs/data
      resourcemanager:
         image: apache/hadoop:3.3.6
         hostname: resourcemanager
         command: ["yarn", "resourcemanager"]
         ports:
            - 8088:8088
         env_file:
           - ./config
         volumes:
           - ./test.sh:/opt/test.sh
      nodemanager:
         image: apache/hadoop:3.3.6
         command: ["yarn", "nodemanager"]
         env_file:
           - ./config
   

2. 清理旧数据并重启集群

   • 停止所有集群容器：

     docker-compose down
     

   • 删除DataNode的旧存储目录（若需保留原有NameNode数据，仅删除DataNode目录即可）：

     rm -rf ./datanode1-data ./datanode2-data
     

   • 若需重置整个集群，同时删除NameNode目录：

     rm -rf ./namenode-data ./datanode1-data ./datanode2-data
     

   • 创建挂载目录并设置权限（避免容器内权限不足）：

     mkdir -p ./namenode-data ./datanode1-data ./datanode2-data
     chmod 777 ./namenode-data ./datanode1-data ./datanode2-data
     

   • 重新启动集群：

     docker-compose up -d
     

3. 验证修复效果
   启动后通过HDFS WebUI（http://localhost:9870）查看DataNode状态，或执行命令：

   docker-compose exec namenode hdfs dfsadmin -report
   

   确认所有DataNode正常注册并运行。

内容的提问来源于stack exchange，提问作者Vladimir