Docker容器中PySpark运行时AWS Java SDK包下载大小不匹配报错

在Docker容器中运行PySpark脚本，通过s3a协议将数据写入AWS时，出现依赖包下载失败，最终导致Java网关进程退出的问题。

执行docker run后的初始输出 #

:: loading settings :: url = jar:file:/usr/local/lib/python3.9/site-packages/pyspark/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
org.apache.hadoop#hadoop-aws added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-b351103a-0cb0- 46f6-9894-49096e91d74a;1.0
    confs: [default]
    found org.apache.hadoop#hadoop-aws;3.3.4 in central
    found com.amazonaws#aws-java-sdk-bundle;1.12.262 in central
    found org.wildfly.openssl#wildfly-openssl;1.0.7.Final in central
downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar ...
    [SUCCESSFUL ] org.apache.hadoop#hadoop-aws;3.3.4!hadoop-aws.jar (104510ms)
downloading https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-  bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar ...
downloading https://repo1.maven.org/maven2/org/wildfly/openssl/wildfly-openssl/1.0.7.Final/wildfly-openssl-1.0.7.Final.jar ...
    [SUCCESSFUL ] org.wildfly.openssl#wildfly-openssl;1.0.7.Final!wildfly- openssl.jar (27243ms)

报错信息 #

依赖包下载失败 #

:: problems summary ::
:::: WARNINGS
[FAILED     ] com.amazonaws#aws-java-sdk-bundle;1.12.262!aws-java-sdk-     bundle.jar: Downloaded file size (9846784) doesn't match expected Content Length (280645251) for https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar. Please retry. (1218943ms)

            [FAILED     ] com.amazonaws#aws-java-sdk-bundle;1.12.262!aws-java-sdk-bundle.jar: Downloaded file size (9846784) doesn't match expected Content Length (280645251) for https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar. Please retry. (1218943ms)

    ==== central: tried

      https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar

            ::::::::::::::::::::::::::::::::::::::::::::::

            ::              FAILED DOWNLOADS            ::

            :: ^ see resolution messages for details  ^ ::

            ::::::::::::::::::::::::::::::::::::::::::::::

            :: com.amazonaws#aws-java-sdk-bundle;1.12.262!aws-java-sdk-bundle.jar

            ::::::::::::::::::::::::::::::::::::::::::::::

最终运行错误 #

RuntimeError: Java gateway process exited before sending its port number

相关文件内容 #

Dockerfile #

ARG IMAGE_VARIANT=slim-buster
ARG OPENJDK_VERSION=8
ARG PYTHON_VERSION=3.9.8

FROM python:${PYTHON_VERSION}-${IMAGE_VARIANT} AS py3
FROM openjdk:${OPENJDK_VERSION}-${IMAGE_VARIANT}

COPY --from=py3 / /

ARG PYSPARK_VERSION=3.2.0
RUN pip --no-cache-dir install pyspark==${PYSPARK_VERSION} pandas datetime boto3   botocore uuid

COPY /housing_prices_2022_10.csv /app/housing_prices_2022_10.csv
ADD script.py /

CMD [ "python", "./script.py" ]

Python脚本 #

import os
from pyspark.sql import SparkSession

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""


spark = SparkSession \
    .builder \
    .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.3.4") \
    .getOrCreate()

complete_raw_path = "/app/housing_prices_2022_10.csv"
curated_path = "s3a://your-bucket/path/to/curated"

housing_prices= spark.read.option("header", "true").csv(complete_raw_path)

housing_prices.distinct()

housing_prices.write.parquet( curated_path,
                        mode = "overwrite")

解决方案 #

方案1：镜像构建时预下载依赖包 #

修改Dockerfile，在构建阶段就下载好所需的依赖jar包，避免运行时动态下载失败：

ARG IMAGE_VARIANT=slim-buster
ARG OPENJDK_VERSION=8
ARG PYTHON_VERSION=3.9.8

FROM python:${PYTHON_VERSION}-${IMAGE_VARIANT} AS py3
FROM openjdk:${OPENJDK_VERSION}-${IMAGE_VARIANT}

COPY --from=py3 / /

ARG PYSPARK_VERSION=3.2.0
ARG HADOOP_AWS_VERSION=3.3.4
ARG AWS_SDK_BUNDLE_VERSION=1.12.262

RUN pip --no-cache-dir install pyspark==${PYSPARK_VERSION} pandas datetime boto3 botocore uuid

# 创建PySpark jars目录并下载依赖包
RUN mkdir -p /usr/local/lib/python3.9/site-packages/pyspark/jars
RUN wget -O /usr/local/lib/python3.9/site-packages/pyspark/jars/hadoop-aws-${HADOOP_AWS_VERSION}.jar https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_AWS_VERSION}/hadoop-aws-${HADOOP_AWS_VERSION}.jar
RUN wget -O /usr/local/lib/python3.9/site-packages/pyspark/jars/aws-java-sdk-bundle-${AWS_SDK_BUNDLE_VERSION}.jar https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${AWS_SDK_BUNDLE_VERSION}/aws-java-sdk-bundle-${AWS_SDK_BUNDLE_VERSION}.jar
RUN wget -O /usr/local/lib/python3.9/site-packages/pyspark/jars/wildfly-openssl-1.0.7.Final.jar https://repo1.maven.org/maven2/org/wildfly/openssl/wildfly-openssl/1.0.7.Final/wildfly-openssl-1.0.7.Final.jar

COPY /housing_prices_2022_10.csv /app/housing_prices_2022_10.csv
ADD script.py /

CMD [ "python", "./script.py" ]

方案2：更换Maven镜像源提升下载稳定性 #

修改SparkSession配置，使用更稳定的Maven镜像源：

spark = SparkSession \
    .builder \
    .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.3.4") \
    .config("spark.jars.repositories", "https://maven.aliyun.com/repository/public") \
    .getOrCreate()

方案3：匹配PySpark与Hadoop依赖版本 #

PySpark 3.2.0默认适配Hadoop 3.2.0版本，使用对应版本的依赖可以减少兼容问题：

spark = SparkSession \
    .builder \
    .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.2.0") \
    .getOrCreate()

额外注意事项 #

• 确保容器网络正常，能访问Maven仓库和AWS S3服务
• 检查AWS密钥配置是否正确，避免S3写入权限错误
• 运行容器时可适当增加内存资源，避免大文件下载时内存不足

内容的提问来源于stack exchange，提问作者paul773