Databricks Runtime 10.4环境下无法加载spark-avro JAR包求助

升级集群上的Databricks-Connect运行时至10.4后，执行以下代码时出现加载spark-avro失败的错误：

from pyspark.sql import SparkSession
spark = SparkSession.builder.config("spark.jars.packages", "org.apache.spark:spark-avro_2.12:3.3.0").getOrCreate()

错误日志片段：

The jars for the packages stored in: C:\Users\lazlo\.ivy2\jars
org.apache.spark#spark-avro_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-dc011dfd-9d25-4d6f-9d0e-354626e7c1f8;1.0
    confs: [default]
    found org.apache.spark#spark-avro_2.12;3.3.0 in central
    found org.tukaani#xz;1.8 in central
    found org.spark-project.spark#unused;1.0.0 in central
:: resolution report :: resolve 156ms :: artifacts dl 4ms
    :: modules in use:
    org.apache.spark#spark-avro_2.12;3.3.0 from central in [default]
    org.spark-project.spark#unused;1.0.0 from central in [default]
    org.tukaani#xz;1.8 from central in [default]
    ---------------------------------------------------------------------
    |                  |            modules            ||   artifacts   |
    |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
    ---------------------------------------------------------------------
    |      default     |   3   |   0   |   0   |   0   ||   3   |   0   |
    ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-dc011dfd-9d25-4d6f-9d0e-354626e7c1f8
    confs: [default]
    0 artifacts copied, 3 already retrieved (0kB/5ms)
22/08/16 13:15:57 WARN Shell: Did not find winutils.exe: {}

最终抛出异常：

Traceback (most recent call last):
  File "C:/Aifora/repositories/test_poetry/tmp_jars.py", line 4, in <module>
    spark = SparkSession.builder.config("spark.jars.packages", "org.apache.spark:spark-avro_2.12:3.3.0").getOrCreate()
  File "C:\Users\lazlo\AppData\Local\pypoetry\Cache\virtualenvs\test-poetry-vvodToDL-py3.8\lib\site-packages\pyspark\sql\session.py", line 229, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "C:\Users\lazlo\AppData\Local\pypoetry\Cache\virtualenvs\test-poetry-vvodToDL-py3.8\lib\site-packages\pyspark\context.py", line 400, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "C:\Users\lazlo\AppData\Local\pypoetry\Cache\virtualenvs\test-poetry-vvodToDL-py3.8\lib\site-packages\pyspark\context.py", line 147, in __init__
    self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
  File "C:\Users\lazlo\AppData\Local\pypoetry\Cache\virtualenvs\test-poetry-vvodToDL-py3.8\lib\site-packages\pyspark\context.py", line 210, in _do_init
    self._jsc = jsc or self._initialize_context(self._conf._jconf)
  File "C:\Users\lazlo\AppData\Local\pypoetry\Cache\virtualenvs\test-poetry-vvodToDL-py3.8\lib\site-packages\pyspark\context.py", line 337, in _initialize_context
    return self._jvm.JavaSparkContext(jconf)
  File "C:\Users\lazlo\AppData\Local\pypoetry\Cache\virtualenvs\test-poetry-vvodToDL-py3.8\lib\site-packages\py4j\java_gateway.py", line 1568, in __call__
    return_value = get_return_value(
  File "C:\Users\lazlo\AppData\Local\pypoetry\Cache\virtualenvs\test-poetry-vvodToDL-py3.8\lib\site-packages\py4j\protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.

环境信息 #

• Windows 11 系统
• 使用 Poetry 管理依赖，pyproject.toml 配置如下：

[tool.poetry]
name = "test_poetry"
version = "1.37.5"
description = ""
authors = [
     "lazloo xp <lazloo.xp@xxx.com>",
 ]

[[tool.poetry.source]]
name = "xxx_nexus"
url = "https://nexus.infrastructure.xxxx.net/repository/pypi-all/simple/"
default = true

[tool.poetry.dependencies]
python = "==3.8.*"
databricks-connect = "^10.4"

1. 匹配 spark-avro 与 Databricks Runtime 版本 #

Databricks-Connect 10.4 对应集群的 Spark 版本为 3.2.1，你指定的 spark-avro_2.12:3.3.0 是 Spark 3.3.x 版本的包，版本不兼容导致加载失败。

修改代码中的包版本为兼容的 3.2.1：

from pyspark.sql import SparkSession
spark = SparkSession.builder.config("spark.jars.packages", "org.apache.spark:spark-avro_2.12:3.2.1").getOrCreate()

2. 清理本地 Ivy 缓存 #

之前下载的不兼容版本可能残存在缓存中，导致加载异常：

• 删除目录 C:\Users\lazlo\.ivy2\jars 下的所有文件
• 重新运行代码，让系统重新下载兼容版本的 JAR 包

3. 确认 Databricks-Connect 配置正确性 #

执行以下命令检查配置是否正确，确保集群地址、令牌等信息无误，且集群 Runtime 版本确实为 10.4：

databricks-connect configure

4. 处理 winutils.exe 警告（非直接错误，可选优化） #

虽然该警告不是加载失败的直接原因，但可通过以下步骤消除：

• 下载对应 Hadoop 3.3.1 版本的 winutils.exe
• 创建目录 C:\hadoop\bin，将 winutils.exe 放入其中
• 系统环境变量中添加 HADOOP_HOME=C:\hadoop

内容的提问来源于stack exchange，提问作者Lazloo Xp