首次运行MapReduce程序遇输出文件权限问题求助

Let’s walk through the common fixes for the permission problem you’re facing when running your MapReduce streaming job. Here’s what to check and adjust:

1. Correct the Output Path (Critical!) #

Hadoop MapReduce expects the -output argument to be a directory, not a file. Your current command specifies /mayank/output/res.txt as the output path, which is invalid—Hadoop can’t write results directly to a file. Instead, use a directory name, e.g., /mayank/output/res:

yarn jar /usr/hdp/2.3.4.0-3485/hadoop-mapreduce/hadoop-streaming.jar \
-input mayank/data/upx/wc_data.txt \
-output /mayank/output/res \
-file /home/mykishore231087/d/wordcount_mapper.py \
-file /home/mykishore231087/d/wordcount_reducer.py \
-mapper "wordcount_mapper.py" \
-reducer "wordcount_reducer.py"

Note: I also switched to yarn jar as suggested in the warning, which is the proper way to launch YARN-based MapReduce jobs.

2. Verify HDFS Directory Permissions #

You need write access to the /mayank/output directory in HDFS. Here’s how to check and fix this:

• First, check the permissions of the parent directory:

  hdfs dfs -ls /mayank
  

• If /mayank/output doesn’t exist, create it with your user ownership:

  hdfs dfs -mkdir -p /mayank/output
  hdfs dfs -chown mykishore231087:mykishore231087 /mayank/output
  

• If the directory already exists, ensure you have write permissions:

  hdfs dfs -chmod u+w /mayank/output
  

3. Ensure Local Scripts Have Execute Permissions #

Hadoop needs to run your mapper and reducer scripts, so make sure they’re executable:

chmod +x /home/mykishore231087/d/wordcount_mapper.py
chmod +x /home/mykishore231087/d/wordcount_reducer.py

Also, notice in the revised command above, I used just wordcount_mapper.py instead of the full path—when you use -file, Hadoop copies the script to the job’s working directory, so you can reference it by filename directly.

4. Clean Up Previous Output (If Needed) #

If you tried running the job before and HDFS created a partial output directory, you’ll need to delete it first (Hadoop won’t overwrite existing directories):

hdfs dfs -rm -r /mayank/output/res

After making these adjustments, re-run your job, and the permission issues should be resolved.

内容的提问来源于stack exchange，提问作者Mayank Kishore