升级Spark至2.2.1创建Spark Session时遇UnsafeOutput错误

Hey there, sorry to hear you're stuck on this Kryo error right after taking over maintenance of Hydrograph—let's get this sorted out quickly.

What's Causing This? #

Spark 2.2.1 relies on a specific version of the Kryo serialization library under the hood. When you upgraded Spark, your project's existing dependencies likely don't include the correct Kryo version that provides the UnsafeOutput class, or there's a version conflict between old dependencies and the one Spark 2.2.1 expects. That's why the error pops up at line 89 of HydrographRuntime.scala—the code is trying to use a Kryo class that's missing from your classpath.

Step-by-Step Solution #

The fix is straightforward: add the compatible Kryo dependency to your project's build.gradle file. Here's how:

1. Navigate to the build.gradle file for the hydrograph.engine.spark module (or the root project's build file if dependencies are declared there).

2. Add the following line to your dependencies block:

   dependencies {
       // Keep your existing dependencies here
       implementation 'com.esotericsoftware:kryo:4.0.2'
   }
   

   Note: Spark 2.2.1 officially uses Kryo 4.0.2, so this version should resolve the missing class issue. If you still see conflicts later, you can adjust the version to match what Spark 2.2.1 pulls in.

3. After adding the dependency, run a clean build to refresh your classpath:

   ./gradlew clean build
   

Extra Troubleshooting Tips (For New Maintainers) #

• If you still run into dependency conflicts, use Gradle's dependency insight tool to track down conflicting versions:

  ./gradlew dependencyInsight --dependency kryo
  

  This will show you all the places Kryo is being pulled in, and you can exclude incompatible versions using Gradle's exclude syntax if needed.
• When upgrading Spark versions, always cross-check Spark's official documentation for its transitive dependency versions—this helps avoid mismatches like this in the future.
• Ensure that when you package your Spark job, the Kryo dependency is included in the final JAR (using implementation instead of compileOnly should handle this).

内容的提问来源于stack exchange，提问作者ashutosh mittal