Cloudera终端上传本地CSV至HDFS报错求助（新手）

Hey there! As a second-year student new to these tools, it’s totally normal to hit snags like this—let’s walk through the most common fixes step by step.

1. Double-Check Your Local File Path (The #1 Culprit!) #

First, make sure you’re pointing to the exact absolute path of your CSV file. Relative paths can trip you up if your terminal’s working directory isn’t where the file lives.

• To confirm the file exists locally:

  ls -l /full/path/to/your/file.csv
  

  You should see the file listed with readable permissions (look for r in the first set of characters, e.g., -rw-r--r--).

• When uploading, use the full local path instead of a relative one. For example:

  # Correct: absolute local path
  hadoop fs -put /home/diana/your_data.csv /user/diana/hdfs_target/
  # Avoid: relative path (only works if terminal is in the same folder as the CSV)
  hadoop fs -put your_data.csv /user/diana/hdfs_target/
  

2. Verify the HDFS Target Directory Exists & You Have Permissions #

HDFS won’t automatically create parent directories unless you tell it to.

• First, create the target directory (use -p to make parent folders if needed):

  hadoop fs -mkdir -p /user/diana/hdfs_target/
  

• Check if the directory exists:

  hadoop fs -ls /user/diana/
  

• Confirm you have write permissions to the target:

  hadoop fs -test -w /user/diana/hdfs_target/
  

  If this command returns nothing, you’re good to go—any error means you need to adjust permissions (ask your lab instructor if you don’t have permission to modify HDFS permissions).

3. Ensure HDFS Services Are Running Correctly #

Even if you turned off safe mode, other HDFS components might be misbehaving.

• Check if NameNode and DataNode are up (in Cloudera Manager, look for green status icons). Alternatively, run this in the terminal:

  hdfs dfsadmin -report
  

  Look for lines like Live datanodes (1) to confirm nodes are active.

• Double-check safe mode is truly disabled:

  hdfs dfsadmin -safemode get
  

  It should return Safe mode is OFF.

4. Fix Common Command Mistakes #

• Case sensitivity: HDFS paths are case-sensitive! /user/Diana and /user/diana are two different directories—make sure your target path matches exactly what you created.
• Typos: Double-check file names and directory paths for spelling errors (e.g., data.csv vs dat.csv, /user/hadoop vs /user/hadooop).
• Try copyFromLocal instead: This command is identical to put, but sometimes works if there’s a quirk with your Cloudera setup:

  hadoop fs -copyFromLocal /home/diana/your_data.csv /user/diana/hdfs_target/
  

5. Verify the Upload Worked #

After running the upload command, confirm the file is in HDFS:

hadoop fs -ls /user/diana/hdfs_target/

You should see your CSV file listed here.

Quick Tip for Pig Processing #

Once the file is in HDFS, make sure your Pig script uses the full HDFS path, like:

raw_data = LOAD '/user/diana/hdfs_target/your_data.csv' USING PigStorage(',');

内容的提问来源于stack exchange，提问作者DianaS19