Ubuntu虚拟机Hadoop 3.3.6运行PageRank任务报错求助

问题背景 #

在Ubuntu虚拟机部署的Hadoop 3.3.6环境中运行PageRank实验任务，执行命令后出现java.lang.RuntimeException错误，重新下载Hadoop后问题仍未解决。

操作命令 #

初始执行命令 #

hadoop jar Home/hadoop-3.3.6/share/hadoop/tools/lib/hadoop-streaming-3.3.6.jar -input /inputs7/input3.txt -output /inputs7/input3.txt -mapper mapper1.py -reducer reducer1.py -file /home/amina01/Documents/mapper1.py -file /home/amina01/Documents/mapper1.py

重新执行命令 #

amina01@amina01-VirtualBox:~/Documents$ hdfs dfs -mkdir -p /user/amina01
amina01@amina01-VirtualBox:~/Documents$ hdfs dfs -put -f /home/amina01/Documents/input.txt
amina01@amina01-VirtualBox:~/Documents$ hdfs dfs -put -f /home/amina01/Documents/input.txt /user/amina01
amina01@amina01-VirtualBox:~/Documents$ hdfs dfs -mkdir -p /input/output
amina01@amina01-VirtualBox:~/Documents$ python3 driver.py /input/input.txt /input/output /home/amina01/Documents/mapper.py /home/amina01/Documents/reducer.py 5

报错日志（中文翻译版） #

cp: `/input/input.txt': 没有该文件或目录

2024-03-13 23:55:07,241 WARN streaming.StreamJob: -file 参数已废弃，请使用通用参数 -files。

packageJobJar: [/home/amina01/Documents/mapper.py, /home/amina01/Documents/reducer.py, /tmp/hadoop-unjar7254882529784803509/] [] /tmp/streamjob4354537991983504292.jar tmpDir=null

2024-03-13 23:55:08,512 INFO client.DefaultNoHARMFailoverProxyProvider: 正在连接ResourceManager，地址：/0.0.0.0:8032

2024-03-13 23:55:08,885 INFO client.DefaultNoHARMFailoverProxyProvider: 正在连接ResourceManager，地址：/0.0.0.0:8032

2024-03-13 23:55:09,195 INFO mapreduce.JobResourceUploader: 禁用路径的纠删码：/tmp/hadoop-yarn/staging/amina01/.staging/job_1710346569620_0001

2024-03-13 23:55:09,934 INFO mapred.FileInputFormat: 待处理的输入文件总数：0

2024-03-13 23:55:10,056 INFO mapreduce.JobSubmitter: 分片数量：0

2024-03-13 23:55:10,352 INFO mapreduce.JobSubmitter: 提交任务令牌：job_1710346569620_0001

2024-03-13 23:55:10,353 INFO mapreduce.JobSubmitter: 执行使用的令牌：[]

2024-03-13 23:55:10,756 INFO conf.Configuration: 未找到resource-types.xml

2024-03-13 23:55:10,757 INFO resource.ResourceUtils: 无法找到'resource-types.xml'。

2024-03-13 23:55:11,160 INFO impl.YarnClientImpl: 已提交应用：application_1710346569620_0001

2024-03-13 23:55:11,300 INFO mapreduce.Job: 任务跟踪URL：http://amina01-VirtualBox:8088/proxy/application_1710346569620_0001/

2024-03-13 23:55:11,306 INFO mapreduce.Job: 正在运行任务：job_1710346569620_0001

2024-03-13 23:55:21,526 INFO mapreduce.Job: 任务job_1710346569620_0001 未运行在uber模式

2024-03-13 23:55:21,530 INFO mapreduce.Job:  map 0% reduce 0%

2024-03-13 23:55:27,754 INFO mapreduce.Job: 任务ID：attempt_1710346569620_0001_r_000000_0，状态：失败

错误：java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): 子进程执行失败，代码1
 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539)
    at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:454)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2024-03-13 23:55:32,828 INFO mapreduce.Job: 任务ID：attempt_1710346569620_0001_r_000000_1，状态：失败

错误：java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): 子进程执行失败，代码1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539)
    at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:454)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2024-03-13 23:55:38,895 INFO mapreduce.Job: 任务ID：attempt_1710346569620_0001_r_000000_2，状态：失败

错误：java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): 子进程执行失败，代码1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539)
    at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:454)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2024-03-13 23:55:44,938 INFO mapreduce.Job:  map 0% reduce 100%

2024-03-13 23:55:46,023 INFO mapreduce.Job: 任务job_1710346569620_0001 失败，原因：任务task_1710346569620_0001_r_000000失败
任务因子任务失败而终止。失败的Map任务数：0，失败的Reduce任务数：1，被杀死的Map任务数：0，被杀死的Reduce任务数：0

2024-03-13 23:55:46,200 INFO mapreduce.Job: 计数器：7
    任务计数器 
        失败的Reduce任务数=4
        启动的Reduce任务数=4
        所有Map任务占用插槽总时间(ms)=0
        所有Reduce任务占用插槽总时间(ms)=12214
        所有Reduce任务总耗时(ms)=12214
        所有Reduce任务总虚拟核耗时(ms)=12214
        所有Reduce任务总内存耗时(MB·ms)=12507136
2024-03-13 23:55:46,200 ERROR streaming.StreamJob: 任务执行失败！

Streaming命令执行失败！

排查与解决方案 #

1. 修正输入文件路径不匹配问题 #

报错开头提示/input/input.txt不存在，而实际操作中仅将input.txt上传到了/user/amina01目录，导致Hadoop找不到输入文件，进而出现Total input files to process : 0的问题。

解决步骤：

• 在HDFS创建/input目录并上传文件：

  hdfs dfs -mkdir -p /input
  hdfs dfs -put /home/amina01/Documents/input.txt /input/
  

• 或者修改driver.py中的输入路径为/user/amina01/input.txt

2. 修复Python脚本执行问题 #

PipeMapRed.waitOutputThreads(): subprocess failed with code 1表明Python脚本执行出错，需检查以下几点：

• 脚本开头添加shebang行，指定Python解释器：
  在mapper.py和reducer.py第一行加入：

  #!/usr/bin/env python3
  

• 给脚本添加可执行权限：

  chmod +x /home/amina01/Documents/mapper.py /home/amina01/Documents/reducer.py
  

• 本地测试脚本逻辑：

  cat /home/amina01/Documents/input.txt | python3 /home/amina01/Documents/mapper.py | sort | python3 /home/amina01/Documents/reducer.py
  

  确认输出符合预期，排除脚本逻辑错误。

3. 修正Hadoop Streaming参数 #

• 日志提示-file参数已废弃，改用-files参数传递脚本文件
• 初始命令中-output路径与-input路径相同，Hadoop不允许输出目录与输入目录重复，需指定独立的输出目录，且输出目录不能提前存在（Hadoop会自动创建）

4. 验证HDFS路径正确性 #

执行以下命令确认文件存在：

hdfs dfs -ls /input
hdfs dfs -ls /user/amina01

内容的提问来源于stack exchange，提问作者Aminago