如何编译多文件Hadoop MapReduce Java温度统计项目代码？

No worries—compiling multiple separate classes for Hadoop is just a matter of handling the classpath correctly and packaging everything up properly. Let's break this down step by step:

Step 1: Prepare Your Files #

First, put all three .java files (Map.java, Reduce.java, MaxTempDriver.java) into a single directory (let's call it temp-project for example). Make sure you have Hadoop installed and your HADOOP_HOME environment variable set up correctly.

Step 2: Compile the Java Files #

Use javac with Hadoop's classpath to compile all three files at once. This ensures the compiler can find all Hadoop dependencies (like Mapper, Reducer, Job, etc.). Run this command from inside your project directory:

javac -classpath $(hadoop classpath) -d ./classes Map.java Reduce.java MaxTempDriver.java

• The -d ./classes flag tells the compiler to output all compiled .class files into a new classes directory (keeps things clean).
• $(hadoop classpath) automatically pulls in all the necessary Hadoop JARs—no need to manually list every single one!

Step 3: Package the Class Files into a JAR #

Hadoop requires your code to be packaged as a JAR file to run. Create a JAR from the classes directory with this command:

jar cvf max-temp.jar -C classes/ .

• cvf stands for "create", "verbose", "file" (creates the JAR and shows what's being added).
• -C classes/ . switches to the classes directory and adds all its contents to the JAR.

Step 4: Run the Hadoop Job #

Now you can execute your job with the Hadoop command. Replace /path/to/input with your actual input data directory, and /path/to/output with a directory that doesn't exist yet (Hadoop will throw an error if the output directory already exists):

hadoop jar max-temp.jar MaxTempDriver /path/to/input /path/to/output

Quick Fix for Your Reducer Code #

I noticed a small bug in your Reduce.java that might cause incorrect results: the max_temp variable is declared as a class-level field, which means it won't reset between different keys. For example, if the first key's max temp is 25, the next key's max temp will start at 25 instead of Float.MIN_VALUE, leading to wrong values. Fix this by moving max_temp inside the reduce method:

@Override
protected void reduce(Text key, Iterable<FloatWritable> values, Context context) throws IOException, InterruptedException {
    float max_temp = Float.MIN_VALUE; // Reset for each key
    float temp = 0;
    Iterator<FloatWritable> itr = values.iterator();
    while (itr.hasNext()) {
        temp = itr.next().get();
        if (temp > max_temp) {
            max_temp = temp;
        }
    }
    context.write(key, new FloatWritable(max_temp));
}

内容的提问来源于stack exchange，提问作者Đặng Tâm