DeltaLake部署报错：java.lang.NoClassDefFoundError相关问题求助

问题场景 #

在Spark 3.3版本的Dataproc集群上安装Delta Lake 2.3（官方标注兼容Spark 3.3），执行步骤如下：

• Jupyter环境配置：

  Kernel: /opt/conda/miniconda3/bin/python
  Python version: 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:23:14) [GCC 10.4.0]
  PySpark version: 3.4.1
  spark version: 3.3.0
  

• 主节点执行pip install delta-spark==2.3.0
• 下载Delta core jar到Spark jars目录：

  sudo wget https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.3.0/delta-core_2.12-2.3.0.jar -P /usr/lib/spark/jars/
  

• 在/etc/spark/conf/spark-defaults.conf添加配置：

  spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension 
  spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
  

执行写入Delta表代码emp_details.write.format("delta").mode("overwrite").save(delta_path)时触发错误：

Py4JJavaError: An error occurred while calling o90.save.
: com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.sql.delta.storage.DelegatingLogStore$

尝试设置PYSPARK_SUBMIT_ARGS：

os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages io.delta:delta-core_2.12:2.3.0 --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog pyspark-shell'

仍报错：

Py4JJavaError: An error occurred while calling o84.save.
: com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: io/delta/storage/LogStore
Please ensure that the delta-storage dependency is included.

If using Python, please ensure you call `configure_spark_with_delta_pip` or use
`--packages io.delta:delta-core_<scala-version>:<delta-lake-version>`.
See https://docs.delta.io/latest/quick-start.html#python.

More information about this dependency and how to include it can be found here:
https://docs.delta.io/latest/porting.html#delta-lake-1-1-or-below-to-delta-lake-1-2-or-above.

按官方入门文档配置后问题依旧，添加delta-storage jar后出现新错误，求解决办法。

核心问题分析 #

Delta Lake 2.3依赖delta-storage组件，单独安装delta-core jar或仅用pip安装delta-spark会导致依赖缺失；同时Dataproc的PySpark版本（3.4.1）与集群Spark版本（3.3.0）不一致，可能引发兼容性冲突。

分步解决方案 #

1. 统一PySpark与集群Spark版本 #

卸载当前PySpark 3.4.1，安装与集群匹配的3.3.0版本：

pip uninstall -y pyspark
pip install pyspark==3.3.0

2. 完整安装Delta Lake依赖包 #

Delta Lake 2.3需要同时安装delta-core和delta-storage的jar包，且版本必须匹配：

# 下载delta-core和delta-storage jar到Spark jars目录
sudo wget https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.3.0/delta-core_2.12-2.3.0.jar -P /usr/lib/spark/jars/
sudo wget https://repo1.maven.org/maven2/io/delta/delta-storage_2.12/2.3.0/delta-storage_2.12-2.3.0.jar -P /usr/lib/spark/jars/

3. 正确配置Spark Session（Jupyter中） #

在Jupyter notebook开头，使用Delta官方提供的configure_spark_with_delta_pip方法初始化Spark Session，避免手动配置环境变量的冲突：

from pyspark.sql import SparkSession
from delta import configure_spark_with_delta_pip

builder = SparkSession.builder \
    .appName("DeltaLakeTest") \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")

spark = configure_spark_with_delta_pip(builder).getOrCreate()

4. 验证配置生效 #

执行以下代码检查Delta Lake是否正常加载：

# 检查Delta扩展是否启用
print(spark.conf.get("spark.sql.extensions"))
# 创建测试DataFrame并写入Delta表
test_df = spark.createDataFrame([(1, "test")], ["id", "name"])
test_df.write.format("delta").mode("overwrite").save("/tmp/delta_test")
# 读取Delta表验证
read_df = spark.read.format("delta").load("/tmp/delta_test")
read_df.show()

5. 若仍有错误：清理冲突依赖 #

• 删除Spark jars目录中可能存在的其他Delta版本jar包：

  sudo rm /usr/lib/spark/jars/delta-*.jar
  

• 重新安装上述依赖包后，重启Jupyter服务和Spark集群：

  sudo systemctl restart jupyter-server
  sudo systemctl restart spark-master
  

关键注意事项 #

• 确保所有节点（主节点+工作节点）都安装了相同版本的Delta jar包，避免分布式执行时的依赖不一致
• Dataproc集群中，优先通过集群初始化脚本安装Delta Lake，而非仅在主节点操作，确保全集群环境一致

内容的提问来源于stack exchange，提问作者user16798185