无法将DataFrame保存为Delta Table的技术问题求助

搭建环境：Spark 3.5、Scala 2.12、JDK 17.0，通过Docker部署Spark集群+Jupyter Notebook。保存DataFrame为CSV格式正常，但保存为Delta Table时报错：
java.lang.NoSuchMethodError: 'scala.collection.Seq org.apache.spark.sql.types.StructType.toAttributes()

测试代码 #

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("test") \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    .master("spark://spark-master:7077") \
    .getOrCreate()

data = [
    (1, "Alice", 34),
    (2, "Bob", 45),
    (3, "Catherine", 29)
]
columns = ["id", "name", "age"]

df = spark.createDataFrame(data, columns)
df.write.format("delta").mode("overwrite").save("/mnt/cluster/test")

Dockerfile.spark #

FROM bitnami/spark:3.5.0

USER root

# Install prerequisites
RUN apt-get update && apt-get install -y curl

# Specify the Scala version
ENV SCALA_VERSION=2.12

# Copy spark-defaults.conf to the correct location
COPY spark-defaults.conf /opt/bitnami/spark/conf/spark-defaults.conf

# Download and install Delta Lake, Azure Data Lake Storage, and AWS JARs
RUN curl -O https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.4.0/delta-core_2.12-2.4.0.jar \
    && curl -O https://repo1.maven.org/maven2/io/delta/delta-storage/3.2.0/delta-storage-3.2.0.jar \
    && mv delta-core_2.12-2.4.0.jar /opt/bitnami/spark/jars \
    && mv delta-storage-3.2.0.jar /opt/bitnami/spark/jar

Dockerfile.jupyter #

FROM jupyter/pyspark-notebook:x86_64-spark-3.5.0

USER root

RUN apt-get update && apt-get install -y curl

# Specify the Scala version
ENV SCALA_VERSION=2.12

# Download and install necessary JAR files
RUN curl -O https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.4.0/delta-core_2.12-2.4.0.jar && \
    curl -O https://repo1.maven.org/maven2/io/delta/delta-storage/3.2.0/delta-storage-3.2.0.jar && \
    mv delta-core_2.12-2.4.0.jar /usr/local/spark/jars && \
    mv delta-storage-3.2.0.jar /usr/local/spark/jars
    
# Copy requirements.txt and install packages
COPY requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir -r /tmp/requirements.txt

USER jovyan

# Set Spark Master URL
ENV SPARK_MASTER=spark://spark-master:7077

# Start JupyterLab in /mnt/cluster
CMD ["jupyter", "lab", "--notebook-dir=/mnt/cluster"]

spark-defaults-jupyter.conf #

spark.jars                                          /usr/local/spark/jars/delta-core_2.12-2.4.0.jar,/usr/local/spark/jars/delta-storage-3.2.0.jar
spark.sql.extensions                                io.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.spark_catalog                     org.apache.spark.sql.delta.catalog.DeltaCatalog

spark-defaults-spark.conf #

spark.jars                                          jars/delta-core_2.12-2.4.0.jar,jars/delta-storage-3.2.0.jar
spark.sql.extensions                                io.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.spark_catalog                     org.apache.spark.sql.delta.catalog.DeltaCatalog

requirements.txt #

pyspark
jupyterlab
delta-spark

docker-compose.yml #

version: '3'

services:
  spark-master:
    build:
      context: .
      dockerfile: Dockerfile.spark
    container_name: spark-master
    environment:
      - SPARK_MODE=master
    ports:
      - "8080:8080"
      - "7077:7077"
    volumes:
      - C:/cluster:/mnt/cluster
      - ./spark-defaults-spark.conf:/opt/bitnami/spark/conf/spark-defaults.conf
    networks:
      - spark-network

  spark-worker:
    build:
      context: .
      dockerfile: Dockerfile.spark
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
    depends_on:
      - spark-master
    volumes:
      - C:/cluster:/mnt/cluster
    networks:
      - spark-network
    deploy:
      mode: replicated
      replicas: 2

  jupyter:
    build:
      context: .
      dockerfile: Dockerfile.jupyter
    container_name: jupyter-lab
    ports:
      - "8888:8888"
    volumes:
      - C:/cluster:/mnt/cluster
      - ./spark-defaults-jupyter.conf:/usr/local/spark/conf/spark-defaults.conf
    depends_on:
      - spark-master
      - spark-worker
    networks:
      - spark-network

networks:
  spark-network:
    driver: bridge

问题原因与解决方案 #

核心问题 #

Delta Lake版本与Spark 3.5不兼容：Delta Lake 2.4.0适配Spark 3.3，而Spark 3.5需要Delta Lake 3.2.0及以上版本。当前混用delta-core 2.4.0和delta-storage 3.2.0，版本不匹配导致方法缺失；同时Dockerfile.spark中存在路径拼写错误，导致worker节点无法加载delta-storage包。

解决步骤 #

1. 统一Delta Lake版本为3.2.0
   替换所有Delta相关JAR包为适配Spark 3.5的版本：

   • delta-core_2.12-3.2.0.jar
   • delta-storage-3.2.0.jar

2. 修正Dockerfile.spark中的路径错误
   原文件中mv delta-storage-3.2.0.jar /opt/bitnami/spark/jar少了s，改为/opt/bitnami/spark/jars。修改后的Dockerfile.spark：

   FROM bitnami/spark:3.5.0
   
   USER root
   
   # Install prerequisites
   RUN apt-get update && apt-get install -y curl
   
   # Specify the Scala version
   ENV SCALA_VERSION=2.12
   
   # Copy spark-defaults.conf to the correct location
   COPY spark-defaults.conf /opt/bitnami/spark/conf/spark-defaults.conf
   
   # Download and install Delta Lake, Azure Data Lake Storage, and AWS JARs
   RUN curl -O https://repo1.maven.org/maven2/io/delta/delta-core_2.12/3.2.0/delta-core_2.12-3.2.0.jar \
       && curl -O https://repo1.maven.org/maven2/io/delta/delta-storage/3.2.0/delta-storage-3.2.0.jar \
       && mv delta-core_2.12-3.2.0.jar /opt/bitnami/spark/jars \
       && mv delta-storage-3.2.0.jar /opt/bitnami/spark/jars
   

3. 修正Dockerfile.jupyter中的Delta版本
   修改下载JAR的部分：

   # Download and install necessary JAR files
   RUN curl -O https://repo1.maven.org/maven2/io/delta/delta-core_2.12/3.2.0/delta-core_2.12-3.2.0.jar && \
       curl -O https://repo1.maven.org/maven2/io/delta/delta-storage/3.2.0/delta-storage-3.2.0.jar && \
       mv delta-core_2.12-3.2.0.jar /usr/local/spark/jars && \
       mv delta-storage-3.2.0.jar /usr/local/spark/jars
   

4. 更新requirements.txt中的delta-spark版本
   指定适配版本：

   pyspark==3.5.0
   jupyterlab
   delta-spark==3.2.0
   

5. 重新构建镜像并启动服务

   docker-compose down
   docker-compose build --no-cache
   docker-compose up -d
   

内容的提问来源于stack exchange，提问作者numersoz