使用SBT子项目时Scala-Reflect依赖缺失问题排查

我在运行包含多个子项目的SBT项目中的子项目时遇到异常。项目结构为主项目multi_app，以及两个子项目app1和app2，配置代码如下：

val baseDependencies = Seq(
        "org.apache.spark" %% "spark-sql" % "2.4.4" % "provided",
        "org.scalameta" %% "mdoc" % "2.2.14" % "provided"
    )


lazy val commonSettings = Seq(
    organization := ...,
    version := "1.0",
    scalaVersion := "2.12.12",
    crossScalaVersions := Seq("2.11.12"),
    resolvers ++= Seq(...),
  libraryDependencies ++= baseDependencies
)

lazy val testSettings = Seq(
  fork in Test := true,
  javaOptions ++= Seq("-Xms512M", "-Xmx2048M", "-XX:MaxPermSize=2048M", "-XX:+CMSClassUnloadingEnabled"),
  testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-oD"),
)


lazy val multi_app = project
  .in(file("."))
  .settings(name := "multi_app")
  .settings(commonSettings: _*)
  .settings(testSettings: _*)
  .aggregate(app1)
  .aggregate(app2)


lazy val app1 = project
  .in(file("./app1"))
  .settings(commonSettings: _*)
  .settings(testSettings: _*)
  .settings(
    name := "app1",
    mainClass in Compile := Some("sparkapps.multi_app.App1"),
    mainClass in run := Some("sparkapps.multi_app.App1"),
    mainClass in assembly := Some("sparkapps.multi_app.App1")
  )
  .dependsOn(app2)

lazy val app2 = project
  .in(file("./app2"))
  .settings(commonSettings: _*)
  .settings(testSettings: _*)
  .settings(
    name := "app2",
  )

app2仅实现如下对象供app1调用：

object MySharedObject {
    def printWord = println("word")
}

执行sbt "project app1" compile和sbt "project app1" run时，出现如下异常：

sbt "project app1" compile
sbt "project app1" run

[error] (run-main-0) java.lang.NoClassDefFoundError: scala/reflect/api/TypeCreator
[error] java.lang.NoClassDefFoundError: scala/reflect/api/TypeCreator
[error]         at sparkapps.multi_app.App1.main(Main.scala)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]         at java.lang.reflect.Method.invoke(Method.java:498)
[error] Caused by: java.lang.ClassNotFoundException: scala.reflect.api.TypeCreator
[error]         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
[error] stack trace is suppressed; run last Compile / bgRun for the full output
[error] Nonzero exit code: 1
[error] (Compile / run) Nonzero exit code: 1
[error] Total time: 3 s, completed Sep 13, 2022 8:39:22 PM

为何子项目运行时会缺失scala-reflect依赖？按常理该依赖应通过commonSettings中的scalaVersion自动引入，且单项目结构下运行正常。

• 原因分析：
  SBT不会自动引入scala-reflect依赖，哪怕指定了scalaVersion。单项目能正常运行大概率是某个依赖间接引入了scala-reflect，而多子项目结构里没有这个间接依赖，就暴露了问题。另外你配置的baseDependencies里的依赖都是provided scope，运行时不会被加载，自然也没法间接提供scala-reflect。

• 解决办法：
  在baseDependencies里显式添加scala-reflect依赖，确保版本和项目的Scala版本一致：

  val baseDependencies = Seq(
          "org.apache.spark" %% "spark-sql" % "2.4.4" % "provided",
          "org.scalameta" %% "mdoc" % "2.2.14" % "provided",
          "org.scala-lang" % "scala-reflect" % scalaVersion.value
      )
  

  同时可以检查app1的代码，是否有隐式使用反射的逻辑（比如case class的反射操作、依赖反射的库），这是触发scala-reflect依赖需求的直接原因。

内容的提问来源于stack exchange，提问作者mongolol