Dataproc Serverless从2.1升级到2.2后出现BigQuery依赖兼容错误

将Dataproc Serverless版本从2.1升级到2.2后，运行时出现如下错误：

Exception in thread "main" java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: com.google.cloud.spark.bigquery.BigQueryRelationProvider not a subtype
    at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:593)
    at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1244)
    at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1273)
    at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)
    at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)
    at scala.collection.convert.JavaCollectionWrappers$JIteratorWrapper.hasNext(JavaCollectionWrappers.scala:46)
    at scala.collection.StrictOptimizedIterableOps.filterImpl(StrictOptimizedIterableOps.scala:225)
    at scala.collection.StrictOptimizedIterableOps.filterImpl$(StrictOptimizedIterableOps.scala:222)
    at scala.collection.convert.JavaCollectionWrappers$JIterableWrapper.filterImpl(JavaCollectionWrappers.scala:83)
    at scala.collection.StrictOptimizedIterableOps.filter(StrictOptimizedIterableOps.scala:218)
    at scala.collection.StrictOptimizedIterableOps.filter$(StrictOptimizedIterableOps.scala:218)
    at scala.collection.convert.JavaCollectionWrappers$JIterableWrapper.filter(JavaCollectionWrappers.scala:83)
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:629)
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:697)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:208)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:172)
    at com.dia.repositories.BigQueryAllocationRepository.process(BigQueryRepository.java:90)
    at com.dia.services.LoyaltyAllocationLoader.load(Test.java:18)
    at com.dia.LoyaltyAllocationMain.main(TestMain.java:42)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:569)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1032)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1124)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1133)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

代码未做修改，最初使用的依赖如下：

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.13</artifactId>
        <version>3.4.0</version>
        <scope>provided</scope>
        <exclusions>
            <exclusion>
                <groupId>com.google.protobuf</groupId>
                <artifactId>protobuf-java</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>com.google.cloud.spark</groupId>
        <artifactId>spark-bigquery-with-dependencies_2.13</artifactId>
        <version>0.34.0</version>
    </dependency>
</dependencies>

尝试更新依赖到最新版本后，仍出现相同错误：

<dependencies>
    
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.13</artifactId>
        <version>3.5.1</version>
        <scope>provided</scope>
        <exclusions>
            <exclusion>
                <groupId>com.google.protobuf</groupId>
                <artifactId>protobuf-java</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>com.google.cloud.spark</groupId>
        <artifactId>spark-bigquery-with-dependencies_2.13</artifactId>
        <version>0.41.0</version>
    </dependency>
    
</dependencies>

• 核心原因：Dataproc Serverless 2.2内置了Spark 3.5.x版本，而你使用的spark-bigquery-with-dependencies依赖包中包含了与Dataproc内置Spark重复的类。BigQueryRelationProvider类被不同类加载器（应用类加载器和Dataproc内置Spark类加载器）加载，导致JVM认为这是两个不同的类型，从而抛出"not a subtype"错误。

• 解决步骤：

  1. 替换带依赖的BigQuery包：将spark-bigquery-with-dependencies_2.13改为spark-bigquery_2.13，避免把重复的Spark类打包到应用Jar中。with-dependencies版本包含了完整的依赖链，会和Dataproc内置的Spark类冲突。
  2. 对齐Spark版本：确保spark-sql依赖的版本与Dataproc 2.2内置的Spark版本完全一致（Dataproc 2.2对应Spark 3.5.0），并且保持provided范围，避免自行打包Spark核心类。
     修改后的依赖示例：

     <dependencies>
         <dependency>
             <groupId>org.apache.spark</groupId>
             <artifactId>spark-sql_2.13</artifactId>
             <version>3.5.0</version>
             <scope>provided</scope>
         </dependency>
     
         <dependency>
             <groupId>com.google.cloud.spark</groupId>
             <artifactId>spark-bigquery_2.13</artifactId>
             <version>0.41.0</version>
         </dependency>
     </dependencies>
     

  3. 清理打包环境：重新打包前清理本地Maven仓库和项目target目录，确保没有残留的旧版本依赖文件。
  4. 调整类加载顺序（临时方案）：如果上述方法无效，提交Dataproc任务时添加配置spark.driver.userClassPathFirst=true和spark.executor.userClassPathFirst=true，强制优先加载用户提供的类，但优先推荐依赖适配的方案。

内容的提问来源于stack exchange，提问作者Chaos