引入GCP BigQuery依赖致Spark读取失败：Jackson Databind版本冲突

以下是几个经过验证的可行方案，你可以根据自己的构建工具（Maven/Gradle）选择适配的方法：

方案1：强制统一Jackson版本并匹配兼容的Connector版本 #

首先确认你使用的Spark、Scala和BigQuery Connector版本完全兼容（比如Spark 3.3.x + Scala 2.12.x对应Connector 0.30.0版本），然后在构建配置中同时排除Spark和BigQuery依赖中的Jackson组件，再显式引入符合要求的Jackson版本（2.12.x范围内，比如2.12.7），最后通过构建工具强制所有依赖使用这个版本。

Maven配置示例 #

<properties>
    <jackson.version>2.12.7</jackson.version>
    <scala.binary.version>2.12</scala.binary.version>
    <spark.version>3.3.0</spark.version>
    <bigquery.connector.version>0.30.0</bigquery.connector.version>
</properties>

<dependencies>
    <!-- Spark SQL依赖，排除自带Jackson -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
        <exclusions>
            <exclusion>
                <groupId>com.fasterxml.jackson.core</groupId>
                <artifactId>*</artifactId>
            </exclusion>
            <exclusion>
                <groupId>com.fasterxml.jackson.module</groupId>
                <artifactId>*</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <!-- BigQuery Connector依赖，排除自带Jackson -->
    <dependency>
        <groupId>com.google.cloud.spark</groupId>
        <artifactId>spark-bigquery-with-dependencies_${scala.binary.version}</artifactId>
        <version>${bigquery.connector.version}</version>
        <exclusions>
            <exclusion>
                <groupId>com.fasterxml.jackson.core</groupId>
                <artifactId>*</artifactId>
            </exclusion>
            <exclusion>
                <groupId>com.fasterxml.jackson.module</groupId>
                <artifactId>*</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <!-- 显式引入统一版本的Jackson组件 -->
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>${jackson.version}</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-core</artifactId>
        <version>${jackson.version}</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-annotations</artifactId>
        <version>${jackson.version}</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.module</groupId>
        <artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
        <version>${jackson.version}</version>
    </dependency>
</dependencies>

<!-- 强制所有依赖使用指定的Jackson版本 -->
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-enforcer-plugin</artifactId>
            <version>3.0.0</version>
            <executions>
                <execution>
                    <id>enforce</id>
                    <goals>
                        <goal>enforce</goal>
                    </goals>
                    <configuration>
                        <rules>
                            <dependencyConvergence/>
                            <bannedDependencies>
                                <excludes>
                                    <exclude>com.fasterxml.jackson.core:jackson-databind:[2.13,),[0,2.12)</exclude>
                                    <exclude>com.fasterxml.jackson.module:jackson-module-scala_${scala.binary.version}:[2.13,),[0,2.12)</exclude>
                                </excludes>
                            </bannedDependencies>
                        </rules>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Gradle配置示例 #

ext {
    jacksonVersion = '2.12.7'
    scalaBinaryVersion = '2.12'
    sparkVersion = '3.3.0'
    bigQueryConnectorVersion = '0.30.0'
}

dependencies {
    // Spark SQL依赖，排除自带Jackson
    implementation "org.apache.spark:spark-sql_$scalaBinaryVersion:$sparkVersion" {
        exclude group: 'com.fasterxml.jackson.core', module: '*'
        exclude group: 'com.fasterxml.jackson.module', module: '*'
    }

    // BigQuery Connector依赖，排除自带Jackson
    implementation "com.google.cloud.spark:spark-bigquery-with-dependencies_$scalaBinaryVersion:$bigQueryConnectorVersion" {
        exclude group: 'com.fasterxml.jackson.core', module: '*'
        exclude group: 'com.fasterxml.jackson.module', module: '*'
    }

    // 统一引入Jackson组件
    implementation "com.fasterxml.jackson.core:jackson-databind:$jacksonVersion"
    implementation "com.fasterxml.jackson.core:jackson-core:$jacksonVersion"
    implementation "com.fasterxml.jackson.core:jackson-annotations:$jacksonVersion"
    implementation "com.fasterxml.jackson.module:jackson-module-scala_$scalaBinaryVersion:$jacksonVersion"
}

// 强制所有依赖使用指定的Jackson版本
configurations.all {
    resolutionStrategy {
        force "com.fasterxml.jackson.core:jackson-databind:$jacksonVersion"
        force "com.fasterxml.jackson.core:jackson-core:$jacksonVersion"
        force "com.fasterxml.jackson.core:jackson-annotations:$jacksonVersion"
        force "com.fasterxml.jackson.module:jackson-module-scala_$scalaBinaryVersion:$jacksonVersion"
    }
}

方案2：使用不带依赖的BigQuery Connector版本 #

如果你之前用的是spark-bigquery-with-dependencies，换成spark-bigquery_${scala.binary.version}（不带with-dependencies后缀的版本），这个版本不会自带大量第三方依赖，能大幅降低冲突概率。之后手动引入必要的GCP基础依赖，同时保持Jackson版本统一。

Maven配置示例 #

<dependency>
    <groupId>com.google.cloud.spark</groupId>
    <artifactId>spark-bigquery_${scala.binary.version}</artifactId>
    <version>${bigquery.connector.version}</version>
    <exclusions>
        <exclusion>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>*</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<!-- 手动引入GCP核心依赖（按需添加） -->
<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-bigquery</artifactId>
    <version>2.20.0</version>
    <exclusions>
        <exclusion>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>*</artifactId>
        </exclusion>
    </exclusions>
</dependency>

方案3：定位并清理隐藏的依赖冲突 #

用构建工具的依赖树命令，找出哪个依赖还在引入不符合版本的Jackson：

• Maven：mvn dependency:tree | grep jackson
• Gradle：./gradlew dependencies | grep jackson

找到冲突的依赖后，针对性地排除它的Jackson组件，或者通过构建工具强制该依赖使用统一的Jackson版本。

额外注意事项 #

• 确保Scala版本和Spark版本严格兼容（比如Spark 3.x对应Scala 2.12.x，Spark 2.x对应Scala 2.11.x），不要跨版本混用。
• 如果在集群环境运行，要保证集群的Spark版本与本地开发版本一致，避免集群端的依赖覆盖你的配置。
• 执行构建前清理缓存（Maven：mvn clean；Gradle：./gradlew clean），避免旧依赖残留导致的冲突。

内容的提问来源于stack exchange，提问作者Arjun V