升级Apache Flink 1.16至1.20后单元测试报ClassNotFoundException

环境与升级情况 #

• 原部署：Flink 1.16.1 + Java 11，Scala编写Flink任务，当前已切换为Java DataStream API（Scala DataStream API已被Flink弃用）
• 升级后：Flink 1.20.0 + Java 17，已将flink-kafka-connector更新至3.4.0-1.20版本

问题表现 #

单元测试执行时出现如下错误，仅在设置fork in Test := false时触发，设置为true则运行正常：

org.apache.flink.util.FlinkException: Failed to execute job 'my-flink-job'.
[info]     at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2487)
[info]     at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2351)
[info]     at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:68)
[info]     at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2325)
[info]     at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:917)
[info]     at com.my.domain.MySpec.$anonfun$new$3(MySpec.scala:40)
[info]     at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]     at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]     at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]     at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]     ...
[info]     Cause: java.lang.RuntimeException: org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster.
[info]     at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
[info]     at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:75)
[info]     at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
[info]     at java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483)
[info]     at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
[info]     at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1193)
[info]     at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1666)
[info]     at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1633)
[info]     at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
[info]     ...
[info]     Cause: org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster.
[info]     at org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
[info]     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
[info]     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
[info]     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
[info]     at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
[info]     at org.apache.flink.util.MdcUtils.lambda$wrapRunnable$1(MdcUtils.java:64)
[info]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info]     at java.base/java.lang.Thread.run(Thread.java:840)
[info]     ...
[info]     Cause: java.util.concurrent.CompletionException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig
[info]     at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
[info]     at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
[info]     at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770)
[info]     at org.apache.flink.util.MdcUtils.lambda$wrapRunnable$1(MdcUtils.java:64)
[info]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info]     at java.base/java.lang.Thread.run(Thread.java:840)
[info]     ...
[info]     Cause: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig
[info]     at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
[info]     at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114)
[info]     at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
[info]     at org.apache.flink.util.MdcUtils.lambda$wrapRunnable$1(MdcUtils.java:64)
[info]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info]     at java.base/java.lang.Thread.run(Thread.java:840)
[info]     ...
[info]     Cause: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig
[info]     at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
[info]     at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
[info]     at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
[info]     at java.base/java.lang.Class.forName0(Native Method)
[info]     at java.base/java.lang.Class.forName(Class.java:467)
[info]     at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
[info]     at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2034)
[info]     at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1898)
[info]     at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2224)
[info]     at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1733)

已做验证 #

• 确认org.apache.flink:flink-core依赖版本为1.20.0，且存在于类路径中
• 通过依赖树检查，所有flink-core依赖均为1.20.0版本，无版本冲突
• 运行时类路径包含以下文件：

flink-core-1.20.0-tests.jar
flink-core-1.20.0.jar
flink-core-api-1.20.0.jar

需求 #

需在不fork新JVM的情况下运行单元测试，因为使用sbt-jacoco生成测试覆盖率报告，fork模式会破坏报告读写的目录结构

内容的提问来源于stack exchange，提问作者soodankit