Spark 1.6搭配Kryo 2.21出现ArrayList空指针异常求助

Hey there, let's break down this frustrating NullPointerException you're hitting. Your hunch about conflicting Kryo versions (2.21 vs 2.22) is spot-on—version mismatches in serialization libraries often lead to these kinds of weird, hard-to-trace issues. Let's walk through how to confirm and fix this:

First, unpack the stack trace #

The NPE happens at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:234), triggered by Kryo's CollectionSerializer.read() during deserialization. In standard JDK implementations, that line checks the length of the ArrayList's internal elementData array. If that array is null when the method is called, you get the NPE. This suggests Kryo is trying to work with an ArrayList that wasn't properly initialized during deserialization—exactly the kind of problem caused by incompatible serializer versions.

Step 1: Verify Kryo version conflicts in your classpath #

First, confirm if Kryo 2.22 is indeed sneaking into your dependencies:

• For Maven projects: Run this command to list all Kryo-related dependencies:

  mvn dependency:tree | grep kryo
  

• For Gradle projects: Use this instead:

  gradle dependencies | grep kryo
  

Look for any entries pointing to version 2.22. Note which parent dependency is pulling it in—you'll need to exclude that version to force your project to use Kryo 2.21 (the version compatible with Spark 1.6.0).

Step 2: Force Kryo 2.21 and exclude conflicting versions #

Once you've identified the problematic dependency, explicitly pin Kryo 2.21 in your build file and exclude any other versions:

Maven example: #

<dependency>
    <groupId>com.esotericsoftware.kryo</groupId>
    <artifactId>kryo</artifactId>
    <version>2.21</version>
</dependency>

<!-- Exclude Kryo 2.22 from the problematic dependency -->
<dependency>
    <groupId>[offending-group-id]</groupId>
    <artifactId>[offending-artifact-id]</artifactId>
    <version>[offending-version]</version>
    <exclusions>
        <exclusion>
            <groupId>com.esotericsoftware.kryo</groupId>
            <artifactId>kryo</artifactId>
        </exclusion>
    </exclusions>
</dependency>

Gradle example: #

dependencies {
    implementation 'com.esotericsoftware.kryo:kryo:2.21'
    implementation('[offending-group-id]:[offending-artifact-id]:[offending-version]') {
        exclude group: 'com.esotericsoftware.kryo', module: 'kryo'
    }
}

After updating your build file, clear your local dependency cache (e.g., mvn clean install -U for Maven) to ensure you're only pulling in Kryo 2.21.

Step 3: Double-check your Kryo configuration #

Your current config looks okay, but a quick sanity check:

• You're registering ArrayList.class explicitly, which is fine (Spark 1.6.0's KryoSerializer does auto-register some collections, but manual registration won't hurt).
• Ensure your custom classes (VC and DC) don't have any serialization quirks—though the stack trace points to ArrayList, it's worth confirming these classes have valid no-arg constructors or other required serialization hooks.

Step 4: Test after fixing dependencies #

Once you've resolved the version conflict, re-run your Spark job. The NPE should disappear if the version mismatch was the root cause. If it persists, you could add debug logging to trace the deserialization process, but odds are the dependency fix will resolve this.

内容的提问来源于stack exchange，提问作者Amit Kumar