Spark MLlib新手求助：运行RandomForest示例遇ExceptionInInitializerError

Hey there! Let's work through this issue together—since you're new to Spark MLlib in a Play project, there are a few common pitfalls we can check first.

1. First, Fix Your Incomplete Spark MLlib Dependency #

Looking at your build.sbt, the Spark MLlib dependency is cut off ("org.apache.spark" %% "spa..."). This is almost certainly causing missing classes that Spark needs to initialize properly. Update your dependencies to include the full spark-mllib entry:

name := """EnsembleAI"""
version := "1.0-SNAPSHOT"
lazy val root = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.11.7"
libraryDependencies ++= Seq(
  jdbc,
  cache,
  ws,
  "org.scalatestplus.play" %% "scalatestplus-play" % "1.5.1" % Test,
  "org.apache.spark" %% "spark-core" % "2.3.0",
  "org.apache.spark" %% "spark-sql" % "2.3.0",
  "org.apache.spark" %% "spark-mllib" % "2.3.0" // Complete this line
)

Scala 2.11.7 is fully compatible with Spark 2.3.0, so that version pairing is fine.

2. Dig Into the Root Cause of ExceptionInInitializerError #

This error usually means a static initialization block failed somewhere. To get the real issue, you need to look at the Caused by: section in your full stack trace—it will tell you exactly what went wrong (e.g., missing Hadoop dependencies, Akka version conflicts, or misconfigured Spark context).

Common Sub-Issues to Check: #

• Missing Hadoop Dependencies: Spark relies on Hadoop client libraries. If you see errors related to Hadoop classes, add this dependency (match the Hadoop version compatible with Spark 2.3.0):

  "org.apache.hadoop" % "hadoop-client" % "2.7.3" // Use % Provided if deploying to a Spark cluster
  

• Akka Version Conflicts: Both Play and Spark use Akka. Spark 2.3.0 uses Akka 2.4.16, while Play 2.5.x (matching your scalatestplus-play version) uses Akka 2.4.x. If you see Akka-related errors, force the compatible version with:

  dependencyOverrides ++= Seq(
    "com.typesafe.akka" %% "akka-actor" % "2.4.16",
    "com.typesafe.akka" %% "akka-stream" % "2.4.16"
  )
  

3. Run the Spark Example Separately From Play's Web Environment #

Play's web application lifecycle can interfere with Spark's context initialization. Instead of running the example as part of the Play app, create a standalone runner class:

Create src/main/scala/RandomForestRunner.scala:

import org.apache.spark.ml.classification.RandomForestClassificationExample

object RandomForestRunner {
  def main(args: Array[String]): Unit = {
    // Explicitly configure Spark to use local mode (good for testing)
    val spark = org.apache.spark.sql.SparkSession.builder()
      .master("local[*]")
      .appName("RandomForestClassificationExample")
      .getOrCreate()
    
    // Run the example
    RandomForestClassificationExample.main(args)
    
    // Stop Spark when done
    spark.stop()
  }
}

Then run it using SBT:

sbt runMain RandomForestRunner

This avoids any conflicts with Play's internal Akka system or web server.

Final Tips #

• Always check the full stack trace—ExceptionInInitializerError is just a wrapper for the real problem.
• For local testing, using master("local[*]") ensures Spark uses all available cores, which speeds up your example.

内容的提问来源于stack exchange，提问作者avy