Java与Spark升级后Logger类类型转换异常问题求助

将Java应用从Java 17升级至Java 21，同步把Apache Spark升级到3.5.0后，启动时抛出类型转换异常，升级前运行正常。

异常信息 #

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:109)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: java.lang.ClassCastException: class org.apache.logging.slf4j.SLF4JLoggerContext cannot be cast to class org.apache.logging.log4j.core.LoggerContext (org.apache.logging.slf4j.SLF4JLoggerContext and org.apache.logging.log4j.core.LoggerContext are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @20fa23c1)
        at org.apache.spark.util.Utils$.setLogLevel(Utils.scala:2318)
        at org.apache.spark.SparkContext.setLogLevel(SparkContext.scala:399)
        at org.apache.spark.api.java.JavaSparkContext.setLogLevel(JavaSparkContext.scala:673)
        at cz.cuni.matfyz.mminfer.wrappers.MongoDBSchemaLessWrapper.initiateContext(MongoDBSchemaLessWrapper.java:72)
        at cz.cuni.matfyz.mminfer.algorithms.rba.RecordBasedAlgorithm.process(RecordBasedAlgorithm.java:27)
        at cz.cuni.matfyz.mminfer.MMInferOneInAll.main(MMInferOneInAll.java:45)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        ... 5 more

启动命令 #

java --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED  --add-opens=java.base/java.lang.reflect=ALL-UNNAMED  --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -jar target/MM-Infer-One-In-All-1.0-SNAPSHOT.jar C:\Users\alzbe\Documents\mff_mgr\Diplomka\Apps\temp\checkpoint srutkova yelpbusinesssample

POM相关依赖 #

<dependencies>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.12</artifactId>
        <version>3.5.0</version>
        <!--<scope>provided</scope>-->
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>3.5.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
        <version>2.4.13</version>
    </dependency>
    <!--https://mvnrepository.com/artifact/org.mongodb/mongo-java-driver--> 
    <dependency>
        <groupId>org.mongodb</groupId>
        <artifactId>mongo-java-driver</artifactId>
        <version>3.12.10</version>
    </dependency>
    <!--https://mvnrepository.com/artifact/org.mongodb.spark/mongo-spark-connector--> 
    <dependency>
        <groupId>org.mongodb.spark</groupId>
        <artifactId>mongo-spark-connector_2.12</artifactId>
        <version>3.0.1</version>
        <scope>compile</scope>
        <type>jar</type>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>3.8</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter</artifactId>
        <version>5.4.0</version>
        <scope>compile</scope>
    </dependency>
</dependencies>

已尝试更换Spring Boot版本、排除slf4j/log4j依赖，但问题未解决；排除log4j后无法连接MongoDB。

这个异常是日志依赖冲突导致的：Spark 3.5.0尝试将SLF4J的LoggerContext强制转换为Log4j2的LoggerContext，但当前类路径中存在多个日志桥接包，造成了类加载冲突。

1. 统一日志实现，替换Spring Boot默认日志 #

在Spring Boot Starter依赖中排除默认的spring-boot-starter-logging，明确引入spring-boot-starter-log4j2，让整个应用使用Log4j2作为日志实现：

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
    <version>2.4.13</version>
    <exclusions>
        <exclusion>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-logging</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<!-- 引入Log4j2 Starter -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-log4j2</artifactId>
    <version>2.4.13</version>
</dependency>

2. 排除Spark依赖中的冲突日志包 #

Spark 3.5.0自带的SLF4J桥接包会和Log4j2冲突，需要手动排除：

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.12</artifactId>
    <version>3.5.0</version>
    <exclusions>
        <exclusion>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
        <exclusion>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.12</artifactId>
    <version>3.5.0</version>
    <exclusions>
        <exclusion>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
        <exclusion>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
        </exclusion>
    </exclusions>
</dependency>

3. 验证MongoDB日志兼容性 #

Mongo Java Driver 3.12.10使用SLF4J作为日志API，spring-boot-starter-log4j2已经包含SLF4J到Log4j2的桥接包，不会影响MongoDB连接。

4. 升级Spring Boot到兼容Java 21版本（推荐） #

当前使用的Spring Boot 2.4.13不支持Java 21，建议升级到Spring Boot 3.2+版本（3.x最低支持Java 17，3.2+适配Java 21）。升级后可同步将Mongo Spark Connector升级到适配Spark 3.5.0的版本（如10.2.1），进一步减少版本兼容问题。

内容的提问来源于stack exchange，提问作者alzbeta.srutkova