HDFS上传文件提示No such file or directory问题求助

When you run commands with sudo, you’re switching to the root user (or another user if specified), which has a different set of permissions and access context than your regular user account. Here are the most common reasons for this mismatch, along with step-by-step fixes:

1. NFS root_squash is Blocking Root Access (Most Likely Cause) #

If your /home directory is mounted via NFS, the server almost certainly has root_squash enabled by default. This setting maps the root user to an anonymous, unprivileged user on the NFS share—so root can’t access files owned by your regular user account, even though you can see them with your own permissions.

How to Verify: #

Run this command to check if root can see the file:

sudo ls -l /home/user_name/file_name.csv

If this returns "No such file or directory", NFS root_squash is the issue.

Fixes: #

• Option 1: Copy the file to a local (non-NFS) directory first:
  Move the file to /tmp (a local filesystem root can access) then upload to HDFS:

  cp /home/user_name/file_name.csv /tmp/
  sudo hdfs dfs -put /tmp/file_name.csv tempfolder
  rm /tmp/file_name.csv
  

• Option 2: Run the HDFS command as your regular user (if permitted):
  If your user account has write access to the HDFS tempfolder, skip sudo entirely:

  hdfs dfs -put /home/user_name/file_name.csv tempfolder
  

  If you don’t have write access to tempfolder, ask your Hadoop admin to grant permissions, or use the hdfs user directly (see next section).

2. You’re Running the Command as Root Instead of the hdfs User #

Using plain sudo runs the command as root, but HDFS operations are intended to be run by the dedicated hdfs user. Root may have compatibility issues with the HDFS client, even if it can access the local file.

Fix: #

Run the command as the hdfs user instead:

sudo -u hdfs hdfs dfs -put /home/user_name/file_name.csv tempfolder

Before doing this, ensure the hdfs user can read your file:

# Quick (less secure) fix: grant world read access
chmod o+r /home/user_name/file_name.csv

# More secure fix: add hdfs to your user group and grant group read access
sudo usermod -aG your_username hdfs
chmod g+r /home/user_name/file_name.csv

3. SELinux is Blocking Access #

If your system uses SELinux (common on RHEL/CentOS), it may prevent the HDFS process (running as root/hdfs) from accessing files in your home directory due to security context restrictions.

How to Verify: #

Check the SELinux context of your file:

ls -Z /home/user_name/file_name.csv

Look for a context like user_home_t. Then check for recent SELinux denials related to HDFS:

ausearch -m avc -ts recent | grep hdfs

If you see denials referencing user_home_t, SELinux is blocking access.

Fix: #

• Temporary test: Set SELinux to permissive mode (resets on reboot):

  sudo setenforce 0
  

• Permanent fix: Work with your admin to create a custom SELinux policy that allows HDFS access to your home directory files.

4. The File is a Broken/Inaccessible Symbolic Link #

If file_name.csv is a symbolic link, root might not have access to the target file it points to (e.g., the target is in another user’s home directory with restrictive permissions).

How to Verify: #

Check if it’s a symlink and its target:

ls -l /home/user_name/file_name.csv

If it starts with lrwxrwxrwx, it’s a symlink. Run sudo ls -l <target-path> to confirm root can access the target.

Fix: #

Either update the symlink to point to a path root/hdfs can access, or use the actual target path in your hdfs dfs -put command.

内容的提问来源于stack exchange，提问作者Foxlooo