SBT多项目传递依赖错误：SparkSQL依赖无法传递问题

Let's break down your problem and fix it step by step. The core issue here relates to how SBT handles dependency scoping and transitivity, especially with Spark's special Provided scope requirement for cluster deployment.

First, Understand Why Your Current Setup Fails #

When you add a Provided dependency in the logic module, SBT treats it as a dependency that will be available at runtime (e.g., from the Spark cluster) but not included in the compile-time classpath. Even though common has a compile-scoped Spark dependency, SBT prioritizes the Provided declaration in logic over the transitive compile dependency, leaving your logic code without access to Spark classes during compilation.

If you don't declare any Spark dependency in logic but still get the error, double-check your multi-project configuration to ensure logic correctly inherits dependencies from common.

Solution 1: Simple Setup (No Spark Exclusion in Packaging) #

If you don't need to exclude Spark jars from your final package (e.g., running locally), you don't need to add any Spark dependency in logic at all. Just ensure your SBT multi-project setup correctly links logic to common:

// build.sbt
lazy val root = (project in file("."))
  .aggregate(common, logic)

lazy val common = (project in file("common"))
  .settings(
    scalaVersion := "2.11.12", // Match Spark 2.2.1's supported Scala version
    libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.1"
  )

lazy val logic = (project in file("logic"))
  .dependsOn(common) // This inherits all compile-scoped dependencies from common

With this setup, logic will automatically inherit the Spark SQL dependency from common, and compilation should work without errors.

Solution 2: Production Setup (Exclude Spark from Packaging) #

For cluster deployment, Spark jars should not be included in your package (they're provided by the cluster). To use Provided scope while still compiling successfully, you need to tell SBT to include Provided dependencies during compilation:

First, define a shared settings sequence to avoid duplication:

val sparkSharedSettings = Seq(
  scalaVersion := "2.11.12",
  // Declare Spark as Provided
  libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.1" % Provided,
  // Add Provided dependencies to compile-time classpath
  dependencyClasspath in Compile ++= (managedClasspath in Provided).value
)

Then apply these settings to both modules:

lazy val root = (project in file("."))
  .aggregate(common, logic)

lazy val common = (project in file("common"))
  .settings(sparkSharedSettings: _*)

lazy val logic = (project in file("logic"))
  .dependsOn(common)
  .settings(sparkSharedSettings: _*)

This configuration ensures:

• Spark dependencies are marked as Provided so they're excluded from your final package
• SBT includes these dependencies during compilation, so your code can resolve org.apache.spark classes
• logic still inherits code and dependencies from common

Additional Checks #

• Verify your Scala version matches Spark 2.2.1's supported versions (only Scala 2.11.x works; Spark 2.2.1 does not support Scala 2.12)
• Ensure your dependsOn(common) line is correctly placed in the logic project definition
• If you're using the SBT Assembly plugin, confirm it's configured to exclude provided dependencies (it should do this by default)

内容的提问来源于stack exchange，提问作者Grant