MapReduce作业报错：预期org.apache.hadoop.io.Text，实际收到LongWritable

Hey there! Let's work through the key mismatch error you're hitting with your YouTube dataset analysis MapReduce job. From the partial mapper code you shared, here are the most likely fixes to resolve this issue:

1. Fix the Mapper Method Name (Critical!) #

You’ve named your method mapper, but MapReduce’s framework expects the method to be map (exact name, no extra letters). This is a super common typo—if the framework can’t find your custom map method, it’ll fall back to the default implementation, which won’t match the output types you set in your driver class.

Your original code snippet:

public void mapper(LongWritable key,Text value,Context c...

Corrected version:

@Override // Add this annotation to enforce proper method overriding
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
    // Your data parsing logic here
}

The @Override annotation is crucial here—it’ll throw a compile-time error if you misspell the method name or get the signature wrong, saving you runtime headaches.

2. Verify Driver Class Output Type Configuration #

Double-check that your driver class explicitly sets the map output key/value types to match your mapper’s actual output:

// In your driver class
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);

Make sure these lines are present and that the types exactly match the generic parameters in your mapper (Mapper<LongWritable, Text, Text, IntWritable>). A mismatch here (e.g., using LongWritable instead of Text for the key) will trigger the type error you’re seeing.

3. Check Context Write Order #

Ensure you’re writing the correct key-value pair to the context in your map method. Your mapper is supposed to output <Text, IntWritable>, so the write call should be:

context.write(category, one);

If you accidentally swap the order (like context.write(one, category)), the types will be inverted, causing a mismatch with your driver’s configuration.

4. Confirm Correct Mapper Import #

Make sure you’re importing the modern MapReduce Mapper class, not the old API version:

// Correct import (modern API)
import org.apache.hadoop.mapreduce.Mapper;

// Avoid this (old API, different method signatures)
// import org.apache.hadoop.mapred.Mapper;

Mixing old and new APIs will lead to unexpected behavior, including type mismatches.

Full Corrected Mapper Example #

Here’s how your mapper should look with all fixes applied:

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

public class YouTubeDataMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    private static final IntWritable one = new IntWritable(1);
    private Text category = new Text();

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        // Example logic: Parse CSV line to extract YouTube category
        String line = value.toString();
        String[] dataFields = line.split(","); // Adjust delimiter to match your dataset
        
        // Add bounds check to avoid ArrayIndexOutOfBoundsException
        if (dataFields.length > 3) { // Replace with the actual index of your category field
            category.set(dataFields[3]);
            context.write(category, one);
        }
    }
}

Start with fixing the method name first—this is the most probable cause of your error. Then work through the other checks to ensure everything aligns.

内容的提问来源于stack exchange，提问作者Jeet