Spark集成Delta Lake时SQLConf$.PARQUET_FIELD_ID_READ_ENABLED()报错解决

我跟着Delta Lake的快速入门教程尝试用Spark访问Delta Lake，但始终无法成功运行。

我的Maven依赖 #

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>Runner</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>io.delta</groupId>
            <artifactId>delta-core_2.12</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
            <version>3.2.3</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.12</artifactId>
            <version>3.2.3</version>
        </dependency>
        <dependency>
            <groupId>io.delta</groupId>
            <artifactId>delta-core_2.12</artifactId>
            <version>2.2.0</version>
        </dependency>
    </dependencies>

</project>

我的Java代码 #

package org.example;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
public class Main {
    public static void main(String[] args) {
        SparkSession spark = SparkSession
                .builder()
                .appName("Java Spark SQL basic example")
                .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
                .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
                .config("spark.master", "local")
                .getOrCreate();

        Dataset<Row> df = spark.range(0, 5).toDF();
        df.write().format("delta").save("./tmp/delta-table");
        df.show();
    }
}

运行时错误信息 #

Exception in thread "main" java.lang.NoSuchMethodError: 'org.apache.spark.internal.config.ConfigEntry org.apache.spark.sql.internal.SQLConf$.PARQUET_FIELD_ID_READ_ENABLED()'
    at io.delta.sql.DeltaSparkSessionExtension.$anonfun$apply$3(DeltaSparkSessionExtension.scala:88)
    at org.apache.spark.sql.SparkSessionExtensions.$anonfun$buildResolutionRules$1(SparkSessionExtensions.scala:174)
    at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at scala.collection.TraversableLike.map(TraversableLike.scala:286)
    at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
    at scala.collection.AbstractTraversable.map(Traversable.scala:108)
    at org.apache.spark.sql.SparkSessionExtensions.buildResolutionRules(SparkSessionExtensions.scala:174)
    at org.apache.spark.sql.internal.BaseSessionStateBuilder.customResolutionRules(BaseSessionStateBuilder.scala:212)
    at org.apache.spark.sql.internal.BaseSessionStateBuilder$$anon$1.<init>(BaseSessionStateBuilder.scala:187)
    at org.apache.spark.sql.internal.BaseSessionStateBuilder.analyzer(BaseSessionStateBuilder.scala:179)
    at org.apache.spark.sql.internal.BaseSessionStateBuilder.$anonfun$build$2(BaseSessionStateBuilder.scala:357)
    at org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:87)
    at org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:87)
    at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:75)
    at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
    at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:183)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
    at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:183)
    at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:75)
    at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:73)
    at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:65)
    at org.apache.spark.sql.Dataset.<init>(Dataset.scala:205)
    at org.apache.spark.sql.Dataset.<init>(Dataset.scala:211)
    at org.apache.spark.sql.SparkSession.range(SparkSession.scala:550)
    at org.apache.spark.sql.SparkSession.range(SparkSession.scala:529)
    at org.example.Main.main(Main.java:16)

解决思路 #

这个错误的核心原因是Delta Lake与Spark版本不兼容：PARQUET_FIELD_ID_READ_ENABLED这个配置项是Spark 3.3及以上版本才新增的，而你使用的Spark版本是3.2.3，Delta Lake 2.2.0的官方要求是搭配Spark 3.3.x版本，版本不匹配就会出现这种方法找不到的错误。

具体修复方案 #

• 方案一：升级Spark版本到3.3.x（推荐）
  匹配Delta Lake 2.2.0的官方兼容要求，修改pom.xml中的Spark依赖版本为3.3.4（LTS稳定版本）：

  <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.12</artifactId>
      <version>3.3.4</version>
  </dependency>
  <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.12</artifactId>
      <version>3.3.4</version>
  </dependency>
  

• 方案二：降级Delta Lake版本到适配Spark 3.2.x的版本
  Delta Lake 1.2.1是适配Spark 3.2.x系列的稳定版本，修改pom.xml中的Delta依赖，同时删除重复的delta-core依赖项：

  <dependency>
      <groupId>io.delta</groupId>
      <artifactId>delta-core_2.12</artifactId>
      <version>1.2.1</version>
  </dependency>
  

另外需要确认：你的Java版本（11）和Scala版本（2.12）与所选的Spark、Delta版本是兼容的，Spark 3.2/3.3系列都支持Java 11和Scala 2.12，无需额外调整。

内容的提问来源于stack exchange，提问作者Finlay Weber