Scala独立Spark应用打包运行时遇SparkSession$类找不到异常

Hey there! I’ve run into this exact issue when starting out with Spark standalone apps, so let’s break down what’s going wrong and how to fix it.

Key Issues Causing the Error #

• Missing spark-sql dependency: SparkSession lives in the spark-sql module, not just spark-core—your current build config only includes spark-core, so the class isn’t present in your jar.
• Incorrect runtime command: You’re using scala path/to/my/jar to run the app, but Spark apps need to be launched via spark-submit (which handles loading Spark’s core dependencies into the runtime classpath).
• Unspecified dependency scope: If you package Spark’s core libraries into your jar, you can run into conflicts with the Spark environment’s existing libraries—we need to mark these dependencies as provided.

Step-by-Step Fix #

1. Update Your build.sbt
   Replace your current dependency section with this to include spark-sql and set the correct scope:

   name := "Spark_Phase2"
   version := "0.1"
   organization := "bartosz.spark"
   scalaVersion := "2.11.8"
   
   libraryDependencies ++= Seq(
     "org.apache.spark" %% "spark-core" % "2.3.0" % Provided,
     "org.apache.spark" %% "spark-sql" % "2.3.0" % Provided
   )
   

   • The Provided scope tells sbt not to include these libraries in your final jar—they’ll be supplied by the Spark runtime environment when you run the app.
   • Adding spark-sql ensures the SparkSession class is available during compilation and runtime.

2. Re-Package Your App
   Run this command in your project root to rebuild the jar:

   sbt package
   

   Your jar will be generated at target/scala-2.11/Spark_Phase2_2.11-0.1.jar (the exact name matches your project settings).

3. Run the App Correctly with spark-submit
   Forget the scala command—use Spark’s dedicated submit tool instead. Replace com.your.package.MainClass with the full qualified name of your main class (e.g., bartosz.spark.MySparkApp):

   spark-submit --class com.your.package.MainClass target/scala-2.11/Spark_Phase2_2.11-0.1.jar
   

Why This Works #

• Adding spark-sql fills the missing class definition for SparkSession.
• Provided scope avoids dependency bloat and conflicts with the Spark cluster’s existing libraries.
• spark-submit ensures all required Spark dependencies are loaded into the runtime classpath, which the plain scala command doesn’t handle.

内容的提问来源于stack exchange，提问作者Bartors