运行Scala Spark程序时出现scalac编译组件错误求助

问题描述 #

运行Scala Spark程序时触发scalac错误，提示编译sbt组件compiler-interface-2.11.8-62.0失败，相关配置和错误信息如下：

build.sbt配置 #

ThisBuild / version := "0.1.0-SNAPSHOT"

ThisBuild / scalaVersion := "2.11.8"

// Spark Core依赖
ThisBuild / libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.8"

lazy val root = (project in file("."))
  .settings(
    name := "Spark_Practice"
  )

错误堆栈（翻译后） #

scalac: 错误：编译sbt组件'compiler-interface-2.11.8-62.0'失败
sbt.internal.inc.CompileFailed: 编译sbt组件'compiler-interface-2.11.8-62.0'失败
at sbt.internal.inc.AnalyzingCompiler$.handleCompilationError$1(AnalyzingCompiler.scala:436)
at sbt.internal.inc.AnalyzingCompiler$.$anonfun$compileSources$5(AnalyzingCompiler.scala:453)
at sbt.internal.inc.AnalyzingCompiler$.$anonfun$compileSources$5$adapted(AnalyzingCompiler.scala:448)
at sbt.io.IO$.withTemporaryDirectory(IO.scala:490)
at sbt.io.IO$.withTemporaryDirectory(IO.scala:500)
at sbt.internal.inc.AnalyzingCompiler$.$anonfun$compileSources$2(AnalyzingCompiler.scala:448)
at sbt.internal.inc.AnalyzingCompiler$.$anonfun$compileSources$2$adapted(AnalyzingCompiler.scala:440)
at sbt.io.IO$.withTemporaryDirectory(IO.scala:490)
at sbt.io.IO$.withTemporaryDirectory(IO.scala:500)
at sbt.internal.inc.AnalyzingCompiler$.compileSources(AnalyzingCompiler.scala:440)
at org.jetbrains.jps.incremental.scala.local.CompilerFactoryImpl$.org$jetbrains$jps$incremental$scala$local$CompilerFactoryImpl$$getOrCompileInterfaceJar(CompilerFactoryImpl.scala:163)
at org.jetbrains.jps.incremental.scala.local.CompilerFactoryImpl.$anonfun$getScalac$1(CompilerFactoryImpl.scala:61)
at scala.Option.map(Option.scala:242)
at org.jetbrains.jps.incremental.scala.local.CompilerFactoryImpl.getScalac(CompilerFactoryImpl.scala:54)
at org.jetbrains.jps.incremental.scala.local.CompilerFactoryImpl.createCompiler(CompilerFactoryImpl.scala:23)
at org.jetbrains.jps.incremental.scala.local.CachingFactory.$anonfun$createCompiler$3(CachingFactory.scala:24)
at org.jetbrains.jps.incremental.scala.local.Cache.$anonfun$getOrUpdate$2(Cache.scala:20)
at scala.Option.getOrElse(Option.scala:201)
at org.jetbrains.jps.incremental.scala.local.Cache.getOrUpdate(Cache.scala:19)
at org.jetbrains.jps.incremental.scala.local.CachingFactory.createCompiler(CachingFactory.scala:24)
at org.jetbrains.jps.incremental.scala.local.LocalServer.doCompile(LocalServer.scala:43)
at org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:30)
at org.jetbrains.jps.incremental.scala.remote.Main$.compileLogic(Main.scala:209)
at org.jetbrains.jps.incremental.scala.remote.Main$.$anonfun$handleCommand$1(Main.scala:192)
at org.jetbrains.jps.incremental.scala.remote.Main$.decorated$1(Main.scala:182)
at org.jetbrains.jps.incremental.scala.remote.Main$.handleCommand(Main.scala:189)
at org.jetbrains.jps.incremental.scala.remote.Main$.serverLogic(Main.scala:165)
at org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:105)
at org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:577)
at com.facebook.nailgun.NGSession.runImpl(NGSession.java:312)
at com.facebook.nailgun.NGSession.run(NGSession.java:198)

解决方案 #

这类错误多由缓存损坏、JDK版本不兼容或IDE编译逻辑冲突导致，按以下步骤逐一排查：

• 清理sbt缓存：删除本地sbt和ivy2缓存目录，强制重新下载编译组件

  • Linux/macOS：执行 rm -rf ~/.sbt/ 和 rm -rf ~/.ivy2/cache/
  • Windows：删除 C:\Users\<你的用户名>\.sbt 和 C:\Users\<你的用户名>\.ivy2\cache 目录

• 切换兼容JDK版本：Scala 2.11.x最高支持JDK 8，错误日志显示当前使用JDK 9+，需切换到JDK 8

  • 在IDEA中打开File > Project Structure > Project，将SDK设置为JDK 8，语言级别设为8

• 改用sbt原生编译：禁用IDEA的JPS增量编译，避免逻辑冲突

  • 打开IDEA设置File > Settings > Build, Execution, Deployment > Build Tools > sbt
  • 勾选Use sbt shell for build and import，重启IDE后重新编译

• 指定兼容sbt版本：在项目根目录创建project/build.properties文件，指定适配Scala 2.11.8的sbt版本（推荐0.13.18）

  sbt.version=0.13.18
  

• 重新导入项目：清理缓存并修改配置后，执行File > Sync Project with sbt Files，等待依赖重新下载完成后再编译

内容的提问来源于stack exchange，提问作者Bhavesh Sonawane