迁移至Java17+SpringBoot3.2.5时Spark4.0-preview1遇NoClassDefFoundError求助

我们正尝试将现有Spark分析引擎迁移至Java 17与SpringBoot 3.2.5环境，使用最新的Spark v4.0-preview版本，但创建SparkSession对象时出现错误。

创建SparkSession的代码：

this.sparkSession = SparkSession.builder()
                        .appName("Spark Session")
                        .master("local[*]")
                        .getOrCreate();

错误堆栈信息：

java.lang.NoClassDefFoundError: jakarta/servlet/SingleThreadModel
    at org.sparkproject.jetty.servlet.ServletHolder.setServlet(ServletHolder.java:173) ~[spark-core_2.13-4.0.0-preview1.jar:4.0.0-preview1]
    at org.sparkproject.jetty.servlet.ServletHolder.<init>(ServletHolder.java:120) ~[spark-core_2.13-4.0.0-preview1.jar:4.0.0-preview1]
    at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:121) ~[spark-core_2.13-4.0.0-preview1.jar:4.0.0-preview1]
    at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:69) ~[spark-core_2.13-4.0.0-preview1.jar:4.0.0-preview1]
    at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:248) ~[spark-core_2.13-4.0.0-preview1.jar:4.0.0-preview1]
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:515) ~[spark-core_2.13-4.0.0-preview1.jar:4.0.0-preview1]
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2965) ~[spark-core_2.13-4.0.0-preview1.jar:4.0.0-preview1]
    at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1122) ~[spark-sql_2.13-4.0.0-preview1.jar:4.0.0-preview1]
    at scala.Option.getOrElse(Option.scala:201) ~[scala-library-2.13.14.jar:?]
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1116) ~[spark-sql_2.13-4.0.0-preview1.jar:4.0.0-preview1]
    at com.vislesha.jobengine.config.AnalysisExecutionJobConfig.lambda$setSparkSessionTask$0(AnalysisExecutionJobConfig.java:54) ~[classes/:?]
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:388) ~[spring-batch-core-5.1.1.jar:5.1.1]
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:312) ~[spring-batch-core-5.1.1.jar:5.1.1]
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-6.1.6.jar:6.1.6]
    at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:255) ~[spring-batch-core-5.1.1.jar:5.1.1]
    at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82) ~[spring-batch-core-5.1.1.jar:5.1.1]
    at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:369) ~[spring-batch-infrastructure-5.1.1.jar:5.1.1]
    at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:206) ~[spring-batch-infrastructure-5.1.1.jar:5.1.1]
    at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:140) ~[spring-batch-infrastructure-5.1.1.jar:5.1.1]
    at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:240) ~[spring-batch-core-5.1.1.jar:5.1.1]
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:229) ~[spring-batch-core-5.1.1.jar:5.1.1]
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:153) ~[spring-batch-core-5.1.1.jar:5.1.1]
    at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:418) ~[spring-batch-core-5.1.1.jar:5.1.1]
    at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:132) ~[spring-batch-core-5.1.1.jar:5.1.1]
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:317) ~[spring-batch-core-5.1.1.jar:5.1.1]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:157) ~[spring-batch-core-5.1.1.jar:5.1.1]
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) ~[spring-core-6.1.6.jar:6.1.6]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:148) ~[spring-batch-core-5.1.1.jar:5.1.1]
    at org.springframework.batch.core.launch.support.TaskExecutorJobLauncher.run(TaskExecutorJobLauncher.java:59) ~[spring-batch-core-5.1.1.jar:5.1.1]
    at com.vislesha.JobEngineApplication$1.run(JobEngineApplication.java:43) ~[classes/:?]
    at org.springframework.boot.SpringApplication.lambda$callRunner$4(SpringApplication.java:786) ~[spring-boot-3.2.5.jar:3.2.5]
    at org.springframework.util.function.ThrowingConsumer$1.acceptWithException(ThrowingConsumer.java:83) ~[spring-core-6.1.6.jar:6.1.6]
    at org.springframework.util.function.ThrowingConsumer.accept(ThrowingConsumer.java:60) ~[spring-core-6.1.6.jar:6.1.6]
    at org.springframework.util.function.ThrowingConsumer$1.accept(ThrowingConsumer.java:88) ~[spring-core-6.1.6.jar:6.1.6]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:798) ~[spring-boot-3.2.5.jar:3.2.5]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:786) ~[spring-boot-3.2.5.jar:3.2.5]
    at org.springframework.boot.SpringApplication.lambda$callRunners$3(SpringApplication.java:774) ~[spring-boot-3.2.5.jar:3.2.5]
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) [?:?]
    at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357) [?:?]
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:510) [?:?]
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) [?:?]
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) [?:?]
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) [?:?]
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [?:?]
    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) [?:?]
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:774) [spring-boot-3.2.5.jar:3.2.5]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:341) [spring-boot-3.2.5.jar:3.2.5]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1354) [spring-boot-3.2.5.jar:3.2.5]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) [spring-boot-3.2.5.jar:3.2.5]
    at com.vislesha.JobEngineApplication.main(JobEngineApplication.java:32) [classes/:?]
Caused by: java.lang.ClassNotFoundException: jakarta.servlet.SingleThreadModel
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[?:?]
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[?:?]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) ~[?:?]
    ... 50 more

解决方案 #

原因 #

jakarta.servlet.SingleThreadModel在Jakarta Servlet 5.0及以上版本中已被彻底移除，但Spark 4.0-preview依赖的Jetty组件仍在引用这个废弃类，而SpringBoot 3.2.5默认使用的是Jakarta Servlet 6.0，因此出现类找不到的错误。

可选方案 #

• 添加兼容的Servlet API依赖
  在项目的依赖管理中加入保留了SingleThreadModel类的Jakarta Servlet 4.x版本，填补依赖缺口：
  Maven（pom.xml）：

  <dependency>
      <groupId>jakarta.servlet</groupId>
      <artifactId>jakarta.servlet-api</artifactId>
      <version>4.0.4</version>
      <scope>provided</scope>
  </dependency>
  

  Gradle（build.gradle）：

  implementation 'jakarta.servlet:jakarta.servlet-api:4.0.4'
  

• 禁用Spark UI
  如果不需要Spark的Web监控界面，可以直接关闭UI模块，避免加载Jetty相关代码：

  this.sparkSession = SparkSession.builder()
          .appName("Spark Session")
          .master("local[*]")
          .config("spark.ui.enabled", "false")
          .getOrCreate();
  

• 等待Spark正式版修复
  Spark 4.0-preview是预览版本，官方后续发布的正式版应该会适配Jakarta Servlet 5+的环境，可关注Spark官方更新日志。

内容的提问来源于stack exchange，提问作者Raja Chaitanya Rekapalli