Hadoop 3.0本地文件拷贝至HDFS报错求助（含Web端上传异常）

Hey there, let's troubleshoot this HDFS file upload issue you're hitting. Even though your DataNode directory checks out with correct permissions and logs show it started successfully, both CLI and WebUI uploads failing points to a few possible misconfigurations or hidden issues. Let's go through step-by-step fixes:

First, let's confirm all critical services are running as expected:

• Run the jps command in your terminal. You should see NameNode, DataNode, ResourceManager, and NodeManager listed (if YARN is set up). If any of these are missing, restart the corresponding service.
• Check DataNode status with hdfs dfsadmin -report. Look for lines like Live datanodes (1) (or however many you have deployed) and ensure the "Used" / "Available" space values make sense. If DataNodes are marked as Decommissioned or Dead, that's a clear red flag.

Since you've shared core-site.xml and hdfs-site.xml, let's validate the most impactful settings:

• core-site.xml: Confirm the fs.defaultFS value points to your active NameNode (e.g., hdfs://localhost:9000 or hdfs://your-hostname:9000). A wrong address here will break all client communication with HDFS.
• hdfs-site.xml:
  • Ensure dfs.datanode.data.dir matches the path you confirmed (hadoop-3.0.0/hadoop2_data/hdfs/datanode). Use an absolute path here to avoid relative path confusion (e.g., /home/your-user/hadoop-3.0.0/hadoop2_data/hdfs/datanode).
  • Check dfs.replication: If you only have one DataNode, set this to 1 (default is 3). HDFS will fail to upload files if it can't replicate blocks to enough nodes.
  • If dfs.permissions.enabled is set to true, verify the target HDFS directory has write permissions for your user. Run hdfs dfs -ls / to check permissions, and temporarily use hdfs dfs -chmod 777 /your-target-dir to rule out permission blocks (adjust permissions properly later for security).

You mentioned "errors" but didn't share the specific output—this is crucial for pinpointing the issue! For the CLI upload, run your command again and copy the full error log. Examples of common errors and their fixes:

• If you see Connection refused, your client can't reach the NameNode (check port/firewall settings or confirm the NameNode is running).
• If you see Failed to replicate enough blocks, that's directly tied to the dfs.replication setting or dead DataNodes.
• If you see Permission denied, adjust directory permissions as noted above.

For the WebUI upload, note the exact error message shown in the browser (e.g., "Could not create file" or "No live DataNodes available")—this will narrow down the problem quickly.

You checked DataNode logs, but NameNode logs often hold critical details about upload failures (like metadata issues or block allocation problems). Look for logs in your Hadoop installation's logs directory, named something like hadoop-<your-user>-namenode-<your-host>.log. Search for keywords like "upload", "block", or "error" to find relevant entries.

If all above checks pass and you're still stuck, corrupted metadata might be the culprit. Warning: This will erase all existing data in HDFS, so back up any important files first.

1. Stop all Hadoop services:

   stop-dfs.sh
   stop-yarn.sh
   

2. Delete the NameNode metadata directory (find its path in hdfs-site.xml under dfs.namenode.name.dir).
3. Reformat the NameNode:

   hdfs namenode -format
   

4. Restart all services:

   start-dfs.sh
   start-yarn.sh
   

Let me know if you share the exact error messages or configuration details—I can help narrow this down further!

内容的提问来源于stack exchange，提问作者Amit Singla