Docker中PySpark+Delta Lake因企业代理无法下载Jars求助

我在Docker中搭建PySpark+Delta Lake本地开发环境，已确认版本兼容，用Pipenv管理依赖并配置了Artifactory PyPI源。按官方文档搭建单元测试项目，代码通过configure_spark_with_delta_pip初始化SparkSession，但运行pytest时，因企业代理限制，PySpark的ivy2.jar无法连接Maven仓库下载delta-core_2.12;2.0.0报错。尝试设置HTTP_PROXY/HTTPS_PROXY环境变量、在docker-compose.yml中配置spark.jars.repositories均无效。

解决方案方向一：通过企业Artifactory下载Jars #

配置步骤： #

• 修改Spark的Ivy配置文件
  从Docker容器中提取PySpark自带的Ivy配置文件（路径：/usr/local/lib/python3.9/site-packages/pyspark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml），本地修改后挂载回容器，或直接在Dockerfile中添加自定义ivysettings.xml。在配置文件中添加企业Artifactory的Maven仓库：

  <resolvers>
    <ibiblio name="artifactory-maven" root="https://your-company-artifactory-url/maven-public/" m2compatible="true"/>
    <!-- 保留local-m2-cache等本地解析器 -->
  </resolvers>
  

• 初始化SparkSession时指定仓库
  手动配置SparkConf替换默认仓库设置，避免依赖configure_spark_with_delta_pip的自动下载：

  from pyspark.sql import SparkSession
  from delta import *
  from pyspark import SparkConf
  
  spark_conf = (
      SparkConf()
      .set("spark.jars.repositories", "https://your-company-artifactory-url/maven-public/")
      .set("spark.jars.packages", "io.delta:delta-core_2.12:2.0.0")
  )
  
  spark = SparkSession.builder \
      .appName("DeltaTest") \
      .config(conf=spark_conf) \
      .getOrCreate()
  spark = configure_spark_with_delta_pip(spark)
  

• 配置Docker环境变量（如需认证）
  在docker-compose.yml或Dockerfile中添加Artifactory认证信息：

  environment:
    - SPARK_OPTS="--conf spark.jars.repositories=https://your-company-artifactory-url/maven-public/ --conf spark.jars.packages=io.delta:delta-core_2.12:2.0.0"
    - ARTIFACTORY_USER=your-username
    - ARTIFACTORY_PASSWORD=your-password
  

  同时在Ivy配置文件中添加凭证节点：

  <credentials host="your-company-artifactory-url" realm="Artifactory Realm" username="${env.ARTIFACTORY_USER}" passwd="${env.ARTIFACTORY_PASSWORD}"/>
  

解决方案方向二：手动下载Jars后配置 #

操作步骤： #

• 下载所需Jars
  从可访问的Maven仓库（企业内部镜像或离线包）下载delta-core_2.12-2.0.0.jar及其依赖（如delta-storage_2.12-2.0.0.jar）。
• 复制Jars到指定目录
  将Jars复制到PySpark的jars目录（Docker容器内路径：/usr/local/lib/python3.9/site-packages/pyspark/jars/），或通过Dockerfile的COPY指令添加：

  COPY delta-core_2.12-2.0.0.jar /usr/local/lib/python3.9/site-packages/pyspark/jars/
  COPY delta-storage_2.12-2.0.0.jar /usr/local/lib/python3.9/site-packages/pyspark/jars/
  

• 配置SparkSession加载本地Jars
  初始化时直接指定本地Jars路径，跳过远程下载：

  from pyspark.sql import SparkSession
  from delta import *
  
  spark = SparkSession.builder \
      .appName("DeltaTest") \
      .config("spark.jars", "/usr/local/lib/python3.9/site-packages/pyspark/jars/delta-core_2.12-2.0.0.jar,/usr/local/lib/python3.9/site-packages/pyspark/jars/delta-storage_2.12-2.0.0.jar") \
      .getOrCreate()
  # 直接使用DeltaTable等API，无需调用configure_spark_with_delta_pip
  DeltaTable.isDeltaTable(spark, "path/to/delta")
  

完整错误堆栈跟踪 #

:: loading settings :: url = jar:file:/usr/local/lib/python3.9/site-packages/pyspark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
io.delta#delta-core_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-63bca6fa-4932-4c6d-b13f-4a339629fc26;1.0
        confs: [default]
:: resolution report :: resolve 84422ms :: artifacts dl 0ms
        :: modules in use:
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   1   |   0   |   0   |   0   ||   0   |   0   |
        ---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
                module not found: io.delta#delta-core_2.12;2.0.0

        ==== local-m2-cache: tried

          file:/root/.m2/repository/io/delta/delta-core_2.12/2.0.0/delta-core_2.12-2.0.0.pom

          -- artifact io.delta#delta-core_2.12;2.0.0!delta-core_2.12.jar:

          file:/root/.m2/repository/io/delta/delta-core_2.12/2.0.0/delta-core_2.12-2.0.0.jar

        ==== local-ivy-cache: tried

          /root/.ivy2/local/io.delta/delta-core_2.12/2.0.0/ivys/ivy.xml

          -- artifact io.delta#delta-core_2.12;2.0.0!delta-core_2.12.jar:

          /root/.ivy2/local/io.delta/delta-core_2.12/2.0.0/jars/delta-core_2.12.jar

        ==== central: tried

          https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.0.0/delta-core_2.12-2.0.0.pom

          -- artifact io.delta#delta-core_2.12;2.0.0!delta-core_2.12.jar:

          https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.0.0/delta-core_2.12-2.0.0.jar

        ==== spark-packages: tried

          https://repos.spark-packages.org/io/delta/delta-core_2.12/2.0.0/delta-core_2.12-2.0.0.pom

          -- artifact io.delta#delta-core_2.12;2.0.0!delta-core_2.12.jar:

          https://repos.spark-packages.org/io/delta/delta-core_2.12/2.0.0/delta-core_2.12-2.0.0.jar

                ::::::::::::::::::::::::::::::::::::::::::::::

                ::          UNRESOLVED DEPENDENCIES         ::

                ::::::::::::::::::::::::::::::::::::::::::::::

                :: io.delta#delta-core_2.12;2.0.0: not found

                ::::::::::::::::::::::::::::::::::::::::::::::


:::: ERRORS
        Server access error at url https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.0.0/delta-core_2.12-2.0.0.pom (java.net.ConnectException: Connection refused (Connection refused))

        Server access error at url https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.0.0/delta-core_2.12-2.0.0.jar (java.net.ConnectException: Connection refused (Connection refused))

        Server access error at url https://repos.spark-packages.org/io/delta/delta-core_2.12/2.0.0/delta-core_2.12-2.0.0.pom (java.net.ConnectException: Connection refused (Connection refused))

        Server access error at url https://repos.spark-packages.org/io/delta/delta-core_2.12/2.0.0/delta-core_2.12-2.0.0.jar (java.net.ConnectException: Connection refused (Connection refused))


:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: io.delta#delta-core_2.12;2.0.0: not found]

内容的提问来源于stack exchange，提问作者Saugat Mukherjee