Spark集群调用SparkSession.getOrCreate()报JsonMappingException方法不存在

架构老林

2026-8-12

Spark Submit 抛出 NoSuchMethodError：JsonMappingException 构造方法缺失

执行spark-submit提交任务时，调用SparkSession.getOrCreate()方法触发如下异常，本地运行正常，仅Spark集群环境报错：

java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.JsonMappingException.<init>(Ljava/io/Closeable;Ljava/lang/String;)V
    at com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61)
    at com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:46)
    at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
    at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:718)
    at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:62)
    at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala)
    at org.apache.spark.scheduler.EventLoggingListener.initEventLog(EventLoggingListener.scala:89)
    at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:84)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:610)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
    at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
    at com.hiido.server.service.impl.SparkSqlJob.executing(SparkSqlJob.java:56)
    at com.hiido.server.service.impl.SparkSqlJob.main(SparkSqlJob.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:737)

已确认环境版本：Spark核心版本spark_core_2.12-3.2.1，Jackson版本2.12.3，集群Spark版本3.2.1-bin-hadoop2.7，不认可版本冲突的说法，但目前无法解决问题。

项目pom.xml依赖配置：

<properties>
    <java.version>1.8</java.version>
    <geospark.version>1.2.0</geospark.version>
    <spark.compatible.verison>2.3</spark.compatible.verison>
    <!--<spark.version>2.3.4</spark.version>-->
    <spark.version>3.2.1</spark.version>
    <hadoop.version>2.7.2</hadoop.version>
    <geotools.version>19.0</geotools.version>
</properties>
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>


    <!-- https://mvnrepository.com/artifact/io.netty/netty-all -->
    <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty-all</artifactId>
        <version>4.1.68.Final</version>
    </dependency>


    <!--        <dependency>
                <groupId>com.fasterxml.jackson.core</groupId>
                <artifactId>jackson-databind</artifactId>
                <version>2.4.4</version>
            </dependency>-->
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId> jackson-annotations</artifactId>
        <version>2.12.3</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId> jackson-core</artifactId>
        <version>2.12.3</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.12.3</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.codehaus.janino</groupId>
        <artifactId>commons-compiler</artifactId>
        <version>3.0.8</version>
    </dependency>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.12.15</version>
    </dependency>
    <!-- geospark -->
    <dependency>
        <groupId>org.datasyslab</groupId>
        <artifactId>geospark</artifactId>
        <version>${geospark.version}</version>
    </dependency>
    <dependency>
        <groupId>org.datasyslab</groupId>
        <artifactId>geospark-sql_${spark.compatible.verison}</artifactId>
        <version>${geospark.version}</version>
    </dependency>
    <dependency>
        <groupId>org.datasyslab</groupId>
        <artifactId>geospark-viz_${spark.compatible.verison}</artifactId>
        <version>${geospark.version}</version>
    </dependency>
    <!-- geospark -->
    <dependency>
        <groupId>org.apache.sedona</groupId>
        <artifactId>sedona-core-3.0_2.12</artifactId>
        <version>1.1.1-incubating</version>
    </dependency>
    <dependency>
        <groupId>org.apache.sedona</groupId>
        <artifactId>sedona-sql-3.0_2.12</artifactId>
        <version>1.1.1-incubating</version>
    </dependency>
    <dependency>
        <groupId>org.apache.sedona</groupId>
        <artifactId>sedona-viz-3.0_2.12</artifactId>
        <version>1.1.1-incubating</version>
    </dependency>


    <dependency>
        <groupId>org.datasyslab</groupId>
        <artifactId>sernetcdf</artifactId>
        <version>0.1.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
<!--            <version>2.3.4</version>-->
        <version>3.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.12</artifactId>
        <version>3.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.12</artifactId>
        <version>3.2.1</version>
        <!--<scope>provided</scope>-->
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_2.12</artifactId>
        <version>3.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_2.12</artifactId>
        <version>3.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-core</artifactId>
        <version>${hadoop.version}</version>
        <scope>${dependency.scope}</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>${hadoop.version}</version>
        <scope>${dependency.scope}</scope>
    </dependency>
    <dependency>
        <groupId>net.sourceforge.javacsv</groupId>
        <artifactId>javacsv</artifactId>
        <version>2.0</version>
    </dependency>
    <dependency>
        <groupId>org.postgresql</groupId>
        <artifactId>postgresql</artifactId>
        <version>42.2.5</version>
    </dependency>
    <dependency>
        <groupId>org.codehaus.janino</groupId>
        <artifactId>janino</artifactId>
        <version>3.0.8</version>
    </dependency>


    <dependency>
        <groupId>org.geotools</groupId>
        <artifactId>gt-grid</artifactId>
        <version>${geotools.version}</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.locationtech.spatial4j/spatial4j -->
    <dependency>
        <groupId>org.locationtech.spatial4j</groupId>
        <artifactId>spatial4j</artifactId>
        <version>0.8</version>
    </dependency>
    <!-- JTS is essentially only used for polygons. -->
    <!-- https://mvnrepository.com/artifact/org.locationtech.jts/jts-core -->
    <dependency>
        <groupId>org.locationtech.jts</groupId>
        <artifactId>jts-core</artifactId>
        <version>1.18.1</version>
    </dependency>
</dependencies>

解决方案

1. 标记Spark相关依赖为`provided`

集群环境已预装Spark依赖，打包时无需将其打入Jar包，避免依赖版本冲突。修改pom.xml中Spark相关依赖的scope：

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.12</artifactId>
    <version>3.2.1</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.12</artifactId>
    <version>3.2.1</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.12</artifactId>
    <version>3.2.1</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-hive_2.12</artifactId>
    <version>3.2.1</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-mllib_2.12</artifactId>
    <version>3.2.1</version>
    <scope>provided</scope>
</dependency>

2. 强制统一Jackson版本

通过dependencyManagement锁定Jackson三个核心包的版本，确保所有依赖（包括Spring Boot、GeoSpark等）都使用指定的2.12.3版本：

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
            <version>2.12.3</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>2.12.3</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.12.3</version>
        </dependency>
    </dependencies>
</dependencyManagement>

3. 排查并排除冲突依赖

执行mvn dependency:tree命令查看依赖树，找出引入旧版本Jackson的依赖，用exclusions排除：
例如，如果Spring Boot引入了其他版本的Jackson，可在spring-boot-starter依赖中排除：

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
    <exclusions>
        <exclusion>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
        </exclusion>
        <exclusion>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
        </exclusion>
        <exclusion>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
        </exclusion>
    </exclusions>
</dependency>

4. 使用Shade插件重命名Jackson包

如果上述方法无效，可通过maven-shade-plugin将项目中的Jackson包重命名，彻底隔离集群环境的Jackson：

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.2.4</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <relocations>
                            <relocation>
                                <pattern>com.fasterxml.jackson</pattern>
                                <shadedPattern>com.myproject.shaded.fasterxml.jackson</shadedPattern>
                            </relocation>
                        </relocations>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

内容的提问来源于stack exchange，提问作者haobo chu