导入错误：无法从pyspark.sql.utils导入CapturedException（版本兼容问题）

问题场景 #

当使用以下依赖配置时：

pyspark==3.4.0
delta-spark==2.0.0
python==3.11.4

执行代码片段：

from delta import configure_spark_with_delta_pip

出现如下导入错误日志：

ImportError                               Traceback (most recent call last)
Cell In[1], line 6
      3 from enum import Enum
      4 from datetime import datetime
----> 6 from delta import configure_spark_with_delta_pip
      7 from pyspark.sql import SparkSession
      8 from pyspark.sql.types import (ArrayType, BinaryType, BooleanType, ByteType,
      9                                FloatType, IntegerType, ShortType, StringType,
     10                                StructField, StructType, TimestampType)

File /usr/local/lib/python3.11/site-packages/delta/__init__.py:17
      1 #
      2 # Copyright (2021) The Delta Lake Project Authors.
      3 #
   (...)
     14 # limitations under the License.
     15 #
---> 17 from delta.tables import DeltaTable
     18 from delta.pip_utils import configure_spark_with_delta_pip
     20 __all__ = ['DeltaTable', 'configure_spark_with_delta_pip']

File /usr/local/lib/python3.11/site-packages/delta/tables.py:21
      1 #
      2 # Copyright (2021) The Delta Lake Project Authors.
      3 #
   (...)
     14 # limitations under the License.
     15 #
     17 from typing import (
     18     TYPE_CHECKING, cast, overload, Any, Iterable, Optional, Union, NoReturn, List, Tuple
     19 )
---> 21 import delta.exceptions  # noqa: F401; pylint: disable=unused-variable
     22 from delta._typing import (
     23     ColumnMapping, OptionalColumnMapping, ExpressionOrColumn, OptionalExpressionOrColumn
     24 )
     26 from pyspark import since

File /usr/local/lib/python3.11/site-packages/delta/exceptions.py:21
     19 from pyspark import SparkContext
     20 from pyspark.sql import utils
---> 21 from pyspark.sql.utils import (
     22     AnalysisException,
     23     CapturedException,
     24     IllegalArgumentException,
     25     ParseException
     26 )
     27 from py4j.java_gateway import is_instance_of  # type: ignore[import]
     29 if TYPE_CHECKING:

ImportError: cannot import name 'CapturedException' from 'pyspark.sql.utils' (/usr/local/lib/python3.11/site-packages/pyspark/sql/utils.py)

而切换到以下依赖配置时，代码可正常运行：

pyspark==3.2.2
delta-spark==2.0.0
python==3.10

问题原因 #

delta-spark 2.0.0版本的代码中尝试从pyspark.sql.utils导入CapturedException，但这个类在PySpark 3.4.0中已被移除，导致版本不兼容。

解决方案 #

方案1：升级delta-spark到兼容版本 #

delta-spark 2.4.0及以上版本已适配PySpark 3.4.0，修改依赖配置为：

pyspark==3.4.0
delta-spark>=2.4.0
python==3.11.4

执行依赖更新命令：

pip install --upgrade delta-spark

方案2：降级PySpark到兼容版本 #

如果无需升级delta-spark，可将PySpark降级至3.3.x或更低版本（如之前验证可行的3.2.2），依赖配置如下：

pyspark==3.2.2
delta-spark==2.0.0
python==3.11.4

执行依赖更新命令：

pip install pyspark==3.2.2

内容的提问来源于stack exchange，提问作者stevezkw