Scala继承Java泛型类编写Hadoop Mapper遇编译错误求助

Hey there! Let's break down why your Scala port of the Hadoop Mapper is hitting a compiler error when overriding the map method. I’ve stumbled through this exact issue when migrating Hadoop Java code to Scala, so here are the most common culprits and fixes:

1. Mismatched Generic Type Parameters #

Hadoop’s Mapper is a generic class, and Scala’s type system is stricter about explicit type declarations than Java. If you omit the generic parameters when extending Mapper, the compiler won’t recognize your map method as an override—it’ll treat it as a new method instead.

Wrong:

class MyMapper extends Mapper // Missing generic type args

Right:

class MyMapper extends Mapper[LongWritable, Text, Text, IntWritable]

Replace the type parameters with whatever your job uses (KEYIN, VALUEIN, KEYOUT, VALUEOUT).

2. Exact Method Signature & Annotations for Exceptions #

Java’s map method declares checked exceptions (IOException, InterruptedException), but Scala doesn’t enforce checked exceptions natively. To match the parent method’s signature, you need to add @throws annotations to your Scala method. Also, make sure the Context type references the inner class of your specific Mapper generic instance using Scala’s # syntax.

Correct Override Structure:

@throws(classOf[IOException])
@throws(classOf[InterruptedException])
protected override def map(
    key: LongWritable,
    value: Text,
    context: Mapper[LongWritable, Text, Text, IntWritable]#Context
): Unit = {
    // Your map logic here
}

• The protected modifier matches Java’s parent method access level (Scala defaults to public, which would cause a signature mismatch).
• Unit corresponds to Java’s void return type.

3. Accidentally Using the Wrong Hadoop API #

Double-check your imports! Hadoop has two Mapper APIs:

• org.apache.hadoop.mapreduce.Mapper (newer, recommended API used in Hadoop: The Definitive Guide)
• org.apache.hadoop.mapred.Mapper (old, deprecated API)

If you imported the deprecated mapred version, its map method signature is completely different (it uses OutputCollector and Reporter instead of Context), which will definitely trigger an override error. Stick to the mapreduce package.

Full Working Example #

Here’s a complete Scala Mapper port of the classic WordCount example to tie it all together:

import org.apache.hadoop.io.{IntWritable, LongWritable, Text}
import org.apache.hadoop.mapreduce.Mapper
import java.io.IOException
import java.lang.InterruptedException

class WordCountMapper extends Mapper[LongWritable, Text, Text, IntWritable] {
  private val outputWord = new Text()
  private val one = new IntWritable(1)

  @throws(classOf[IOException])
  @throws(classOf[InterruptedException])
  protected override def map(
      key: LongWritable,
      value: Text,
      context: Mapper[LongWritable, Text, Text, IntWritable]#Context
  ): Unit = {
    val line = value.toString.trim
    line.split("\\s+").foreach { token =>
      outputWord.set(token)
      context.write(outputWord, one)
    }
  }
}

Go through these checks one by one—chances are your issue is a small mismatch in generics, exceptions, or imports that’s throwing the compiler off.

内容的提问来源于stack exchange，提问作者Giorgio