运行C++版Hadoop Pipes程序遇空指针错误，求正确运行方法

问题描述 #

我尝试用Hadoop的MapReducer计算多个CSV文件总营收，先从GitHub获取了C++单词计数示例代码练手：

#include <string>

#include  "stdint.h"  // <--- to prevent uint64_t errors!

#include "Pipes.hh"
#include "TemplateFactory.hh"
#include "StringUtils.hh"

class WordCountMapper : public HadoopPipes::Mapper {
 public:
  // Constructor: does nothing
  WordCountMapper(HadoopPipes::TaskContext& context) {}
  // Map function: Receives a line, outputs (word,"1") to reducer.
  void map(HadoopPipes::MapContext& context) {
    // Get line of text
    std::string line = context.getInputValue();
    // Split into words
    std::vector<std::string> words = HadoopUtils::splitString(line, " ");
    // Emit each word tuple (word, "1" )
    for (unsigned int i = 0; i < words.size(); i++) {
      context.emit(words[i], HadoopUtils::toString(1));
    }
  }
};

class WordCountReducer : public HadoopPipes::Reducer {
 public:
  // Constructor: does nothing
  WordCountReducer(HadoopPipes::TaskContext& context) {}
  // Reduce function
  void reduce(HadoopPipes::ReduceContext& context) {
    int count = 0;
    // Get all tuples with the same key, and count their numbers
    while (context.nextValue()) {
      count += HadoopUtils::toInt(context.getInputValue());
    }
    // Emit (word, count)
    context.emit(context.getInputKey(), HadoopUtils::toString(count));
  }
};

int main(int argc, char *argv[]) {
  return HadoopPipes::runTask(
    HadoopPipes::TemplateFactory<WordCountMapper, WordCountReducer>()
  );
}

代码编译通过后，执行以下命令：

bin/mapred pipes -D hadoop.pipes.java.recordreader=true \
             -D hadoop.pipes.java.recordwriter=true \
             -input input/wordcount/sotu_2015.txt   \
             -output output                         \
             -program input/wordcount/wordcount

运行后抛出NullPointerException：

...
java.lang.Exception: java.lang.NullPointerException
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.lang.NullPointerException
    at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:109)
    at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:72)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
2025-01-02 23:31:20,556 INFO mapreduce.Job: Job job_local830471764_0001 running in uber mode : false
2025-01-02 23:31:20,560 INFO mapreduce.Job:  map 0% reduce 0%
2025-01-02 23:31:20,564 INFO mapreduce.Job: Job job_local830471764_0001 failed with state FAILED due to: NA
2025-01-02 23:31:20,567 INFO mapreduce.Job: Counters: 0
Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:875)
    at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:264)
    at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:505)
    at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:520)

我直接解压Hadoop本地运行，未使用Docker，找不到C++相关错误解决方案，求正确运行方式。

解决方案 #

1. 确认-program参数路径有效性 #

这个NullPointerException大多是Hadoop无法定位编译好的C++可执行文件导致的，需按以下方式处理：

• 若使用HDFS存储可执行文件：先将本地编译好的wordcount上传至HDFS指定目录

  hdfs dfs -mkdir -p /user/your_username/input/wordcount
  hdfs dfs -put ./wordcount /user/your_username/input/wordcount/
  

  运行命令中改用HDFS绝对路径：

  -program /user/your_username/input/wordcount/wordcount
  

• 若本地模式运行：-program需指向本地可执行文件的绝对路径，比如/home/your_username/hadoop/input/wordcount/wordcount，同时添加临时目录配置参数：

  -D mapreduce.job.local.dir=/tmp/hadoop-local
  

2. 确保编译时链接正确的Hadoop库 #

编译C++代码必须指定Hadoop头文件路径并链接对应库，示例编译命令（假设Hadoop安装在/opt/hadoop）：

g++ -std=c++11 wordcount.cpp -o wordcount \
  -I/opt/hadoop/include \
  -L/opt/hadoop/lib/native \
  -lhadoop-pipes -lhadoop-utils

64位系统需额外添加-m64参数。

3. 调整Hadoop运行参数 #

• 移除hadoop.pipes.java.recordreader=true和hadoop.pipes.java.recordwriter=true参数，使用C++原生的RecordReader/Writer；
• 修改后的运行命令示例：

  bin/mapred pipes \
    -input /user/your_username/input/wordcount/sotu_2015.txt \
    -output /user/your_username/output \
    -program /user/your_username/input/wordcount/wordcount
  

4. 检查版本兼容性 #

Hadoop Pipes的C++API在不同版本中存在差异，确保示例代码与当前使用的Hadoop版本匹配，建议使用对应版本的官方示例代码。

5. 本地模式运行额外配置 #

本地模式下需确保：

• hadoop-env.sh中已正确设置JAVA_HOME和HADOOP_CLASSPATH；
• 可执行文件拥有执行权限：

  chmod +x wordcount
  

内容的提问来源于stack exchange，提问作者artjom safonoff