Scala模块与Jackson Databind版本不兼容致Spark测试报错求助

问题场景 #

SBT项目使用ScalaTest和Shared Spark Session开展测试工作，近期运行测试时出现Jackson版本冲突错误，导致测试无法正常执行。

错误信息 #

java.lang.ExceptionInInitializerError
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
.....
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.0 requires Jackson Databind version >= 2.10.0 and < 2.11.0
    at com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61)
    at com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:46)

测试代码 #

import org.apache.spark.sql.QueryTest.checkAnswer
import org.apache.spark.sql.Row
import org.apache.spark.sql.test.SharedSparkSession

class SparkTestSpec extends SharedSparkSession  {
  import testImplicits._
  test("join - join using") {
    val df = Seq(1, 2, 3).toDF("int")

    checkAnswer(df, Row(1) :: Row(2) :: Row(3) :: Nil)
  }
}

当前SBT配置 #

ThisBuild / scalaVersion := "2.12.10"
val sparkVersion = "3.1.0"
val scalaTestVersion = "3.2.1"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion % Test,
  "org.apache.spark" %% "spark-sql" % sparkVersion % Test classifier "tests",
  "org.apache.spark" %% "spark-catalyst" % sparkVersion % Test,
  "org.apache.spark" %% "spark-catalyst" % sparkVersion % Test classifier "tests",
  "org.apache.spark" %% "spark-hive" % sparkVersion % Test,
  "org.apache.spark" %% "spark-hive" % sparkVersion % Test classifier "tests",
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-core" % sparkVersion % Test classifier "tests",
  "log4j" % "log4j" % "1.2.17",
  "org.slf4j" % "slf4j-log4j12" % "1.7.30",

  "org.scalatest" %% "scalatest" % scalaTestVersion % Test,
  "org.scalatestplus" %% "scalacheck-1-14" % "3.2.2.0",

)

解决方案 #

冲突原因 #

Spark 3.1.0依赖的Jackson Scala Module版本为2.10.0，该版本要求Jackson Databind版本必须在>=2.10.0且<2.11.0范围内。项目中存在其他依赖（如ScalaTest或其传递依赖）引入了不符合该范围的Jackson版本，导致版本不匹配报错。

解决步骤 #

1. 强制统一Jackson版本：在SBT配置中添加dependencyOverrides，指定所有Jackson相关组件的版本为2.10.0，确保版本一致。
2. 简化依赖声明：移除重复的依赖配置，减少冗余。

修改后的SBT配置 #

ThisBuild / scalaVersion := "2.12.10"
val sparkVersion = "3.1.0"
val scalaTestVersion = "3.2.1"
// 指定Spark兼容的Jackson版本
val jacksonVersion = "2.10.0"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-core" % sparkVersion,
  // 测试相关依赖
  "org.apache.spark" %% "spark-sql" % sparkVersion % Test classifier "tests",
  "org.apache.spark" %% "spark-catalyst" % sparkVersion % Test classifier "tests",
  "org.apache.spark" %% "spark-hive" % sparkVersion % Test classifier "tests",
  "org.apache.spark" %% "spark-core" % sparkVersion % Test classifier "tests",
  "log4j" % "log4j" % "1.2.17",
  "org.slf4j" % "slf4j-log4j12" % "1.7.30",

  "org.scalatest" %% "scalatest" % scalaTestVersion % Test,
  "org.scalatestplus" %% "scalacheck-1-14" % "3.2.2.0" % Test,
)

// 强制统一Jackson所有组件版本
dependencyOverrides ++= Seq(
  "com.fasterxml.jackson.core" % "jackson-core" % jacksonVersion,
  "com.fasterxml.jackson.core" % "jackson-databind" % jacksonVersion,
  "com.fasterxml.jackson.core" % "jackson-annotations" % jacksonVersion,
  "com.fasterxml.jackson.module" %% "jackson-module-scala" % jacksonVersion
)

说明 #

• dependencyOverrides会强制项目中所有依赖（包括传递依赖）使用指定的Jackson版本，彻底解决版本冲突。
• 移除了重复的spark-sql和spark-core在Test scope的普通依赖，因为只需要带classifier "tests"的测试依赖即可满足Shared Spark Session的使用需求。

内容的提问来源于stack exchange，提问作者Vadim