Hadoop MapReduce单词计数任务长时间无进展求助排查

环境与问题概述 #

• 系统：Linux Mint 21（旧笔记本）
• Hadoop版本：2.7.3
• 现状：HDFS命令执行正常，/my_data目录下存在book1.txt文件；启动YARN后提交WordCount任务，命令为hadoop jar /home/nenn/wordcount.jar WordCount /my_data/book1.txt /my_data/output_wordcount，任务持续显示"Running job"超过5分钟无进展，该jar包在学校环境可正常运行。

排查步骤 #

1. 确认YARN集群组件状态 #

执行jps命令，检查ResourceManager、NodeManager是否正常启动：

jps

• 若NodeManager未启动，查看日志文件~/hadoop/logs/yarn-nenn-nodemanager-*.log定位启动失败原因；
• 若ResourceManager异常，重启YARN服务：~/hadoop/sbin/stop-yarn.sh && ~/hadoop/sbin/start-yarn.sh

2. 通过Web UI查看任务细节 #

访问任务跟踪URL http://my-computer-05:8088/proxy/application_1676124615395_0004/，重点查看：

• Map/Reduce任务的进度（是否卡在0%）
• 是否存在任务失败、资源不足的提示
• NodeManager的内存/CPU使用情况（旧笔记本硬件资源不足可能导致任务无法分配）

3. 检查HDFS路径与权限 #

• 确认输出目录/my_data/output_wordcount不存在（Hadoop不允许输出目录已存在），若存在先删除：

hdfs dfs -rm -r /my_data/output_wordcount

• 验证输入文件权限，确保当前用户nenn有读取权限：

hdfs dfs -ls /my_data/book1.txt

4. 排查JDK版本兼容性 #

Hadoop 2.7.3推荐使用JDK 8，Linux Mint 21默认可能搭载更高版本JDK，反射警告大概率是版本不兼容导致。切换到JDK 8后重试任务：

# 假设已安装openjdk-8-jdk
update-alternatives --config java
update-alternatives --config javac

5. 查看任务日志定位异常 #

通过YARN命令导出任务日志，分析Map/Reduce阶段的报错信息：

yarn logs -applicationId application_1676124615395_0004

重点关注内存溢出、依赖缺失、节点通信失败等问题。

相关命令输出 #

HDFS根目录列表 #

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/home/dell/hadoop/share/hadoop/common/lib/hadoop-auth-2.7.3.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Found 2 items
drwxr-xr-x   - nenn supergroup          0 2023-02-11 15:30 /my_data
drwx------   - nenn supergroup          0 2023-02-11 15:21 /tmp

/my_data目录详情 #

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/home/dell/hadoop/share/hadoop/common/lib/hadoop-auth-2.7.3.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
drwxr-xr-x   - nenn supergroup          0 2023-02-11 15:30 /my_data
> -rw-r--r--   1 nenn supergroup    1174876 2023-02-11 15:01 /my_data/book1.txt
drwx------   - nenn supergroup          0 2023-02-11 15:21 /tmp
drwx------   - nenn supergroup          0 2023-02-11 15:21 /tmp/hadoop-yarn
drwx------   - nenn supergroup          0 2023-02-11 15:29 /tmp/hadoop-yarn/staging
drwx------   - nenn supergroup          0 2023-02-11 15:21 /tmp/hadoop-yarn/staging/d

任务提交日志 #

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/home/dell/hadoop/share/hadoop/common/lib/hadoop-auth-2.7.3.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
23/02/11 15:33:55 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
23/02/11 15:33:55 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
23/02/11 15:33:56 INFO input.FileInputFormat: Total input paths to process : 1
23/02/11 15:33:56 INFO mapreduce.JobSubmitter: number of splits:1
23/02/11 15:33:56 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1676124615395_0004
23/02/11 15:33:56 INFO impl.YarnClientImpl: Submitted application application_1676124615395_0004
23/02/11 15:33:56 INFO mapreduce.Job: The url to track the job: http://my-computer-05:8088/proxy/application_1676124615395_0004/
23/02/11 15:33:56 INFO mapreduce.Job: Running job: job_1676124615395_0004

WordCount源码 #

import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

// 定义WordCount主类
public class WordCount {

  // 定义TokenizerMapper类，负责Map阶段处理
  public static class TokenizerMapper
       extends Mapper<Object, Text, Text, IntWritable>{

    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    @Override
    public void map(Object key, Text value, Context context
                    ) throws IOException, InterruptedException {
      StringTokenizer itr = new StringTokenizer(value.toString());
      while (itr.hasMoreTokens()) {
        // 转换为小写并去除非字母数字字符
        word.set(itr.nextToken().toLowerCase().replaceAll("[^a-z 0-9A-Z]",""));
        context.write(word, one);
      }
    }
  }

  // 定义IntSumReducer类，负责Reduce阶段处理
  public static class IntSumReducer
       extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();

    @Override
    public void reduce(Text key, Iterable<IntWritable> values,
                       Context context
                       ) throws IOException, InterruptedException {
      int sum = 0;
      for (IntWritable val : values) {
        sum += val.get();
      }
      result.set(sum);
      context.write(key, result);
    }
  }

  // 主方法，配置并提交任务
  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

内容的提问来源于stack exchange，提问作者marietar