Spark 3.2.1使用spark-testing-base运行单元测试报错排查问询

开发者阿泽

2026-9-3

问题描述

运行Spark单元测试时抛出如下类不兼容异常：

[error] sbt.ForkMain$ForkError: java.lang.IncompatibleClassChangeError: Expected instance not static method org.scalatest.Assertions.assertionsHelper()Lorg/scalatest/Assertions$AssertionsHelper;
[error]     at com.holdenkarau.spark.testing.StreamingSuiteBase.verifyOutput(StreamingSuiteBase.scala:77)
[error]     at com.holdenkarau.spark.testing.StreamingSuiteBase.verifyOutput$(StreamingSuiteBase.scala:61)
[error]     at com.central.spark.aggregation.streaming.BaseAggregatorSuite.verifyOutput(BaseAggregatorSuite.scala:23)
[error]     at com.holdenkarau.spark.testing.StreamingSuiteBase.$anonfun$testOperation$1(StreamingSuiteBase.scala:162)
[error]     at com.holdenkarau.spark.testing.StreamingSuiteBase.$anonfun$testOperation$1$adapted(StreamingSuiteBase.scala:158)
[error]     at com.holdenkarau.spark.testing.StreamingSuiteCommon.withOutputAndStreamingContext(StreamingSuiteCommon.scala:122)
[error]     at com.holdenkarau.spark.testing.StreamingSuiteCommon.withOutputAndStreamingContext$(StreamingSuiteCommon.scala:114)
[error]     at com.central.spark.aggregation.streaming.BaseAggregatorSuite.withOutputAndStreamingContext(BaseAggregatorSuite.scala:23)
[error]     at com.holdenkarau.spark.testing.StreamingSuiteBase.testOperation(StreamingSuiteBase.scala:158)
[error]     at com.holdenkarau.spark.testing.StreamingSuiteBase.testOperation$(StreamingSuiteBase.scala:149)
[error]     at com.central.spark.aggregation.streaming.BaseAggregatorSuite.testOperation(BaseAggregatorSuite.scala:23)
[error]     at com.central.spark.aggregation.streaming.BaseAggregatorSuite.$anonfun$new$1(BaseAggregatorSuite.scala:89)
[error]     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) (truncated)

当前配置

测试依赖配置：

lazy val coreTestDeps = Seq(
  "org.mockito" % "mockito-all" % 1.10.19 % "test",
  "org.scalatest" %% "scalatest" % 3.2.12 % "it,test",
  "net.sf.opencsv" % "opencsv" % 2.3 % "test",
  "org.json4s" %% "json4s-native" % 3.7.0-M11 % "it,test",
  "org.json4s" %% "json4s-jackson" % 3.7.0-M11 % "it,test",
  "org.apache.spark" %% "spark-streaming" % 3.2.1 % "provided" classifier "tests",
  "org.apache.spark" %% "spark-core" % 3.2.1 % "provided" classifier "tests",
  "com.holdenkarau" %% "spark-testing-base" % "3.2.0_1.1.1" % "test",
  "org.elasticsearch.client" % "elasticsearch-rest-high-level-client" % 7.9.3 % "it,test"
)

sbt构建配置：

scalaVersion := "2.12.15",
Test / parallelExecution := false,
Test / fork := true,
javaOptions ++= Seq("-Xms512M", "-Xmx2048M", "-XX:MetaspaceSize=2048M", "-XX:+CMSClassUnloadingEnabled")

已尝试手动降级scalatest到3.0.9等多个版本，问题未解决。

根因分析

该异常是典型的二进制不兼容错误：spark-testing-base 3.2.0_1.1.1编译时依赖的scalatest版本，与运行时类路径中实际加载的scalatest版本方法签名不匹配。
scalatest在3.1到3.2版本迭代时，对Assertions特质内的assertionsHelper成员做了修饰符变更（从静态成员改为实例成员），跨版本调用时就会抛出该异常。
之前手动降级scalatest未生效的核心原因是：没有通过sbt强制锁定版本，依赖仲裁机制仍然选择了高版本scalatest加载，同时未排除spark-testing-base传递进来的内置scalatest依赖，导致类路径下存在多版本scalatest类文件，加载时出现混淆。

解决方案

按以下步骤排查修复：

先确认类路径下的scalatest版本冲突情况，执行命令：
```
sbt "test:dependencyTree" | grep scalatest
```
正常情况下所有scalatest相关依赖（scalatest-core、scalatest-wordspec等子模块）版本必须完全一致。
选择以下任意一种方案对齐版本：
- 方案一（推荐）：升级spark-testing-base到适配scalatest 3.2.x的版本，替换原有spark-testing-base依赖为：
```
"com.holdenkarau" %% "spark-testing-base" % "3.2.1_1.5.3" % "test"
```
  该版本编译时绑定scalatest 3.2系列，与当前使用的3.2.12版本二进制兼容。
- 方案二：保留现有spark-testing-base版本，强制锁定全项目scalatest版本为其兼容的3.1.4，在build.sbt中添加依赖覆盖规则：
```
dependencyOverrides ++= Seq(
  "org.scalatest" %% "scalatest" % "3.1.4" % "it,test"
)
```
清除旧编译缓存后重新运行测试：
```
sbt clean test
```

如果问题仍然存在，可以在测试JVM参数中添加-verbose:class，启动测试后观察日志中org.scalatest.Assertions类的加载路径，确认加载的是预期版本的jar包，排除旧缓存、类路径污染问题。

内容的提问来源于stack exchange，提问作者thebluemagician