寻求可运行PySpark读取S3数据的Dockerfile及故障排查方案

我需要用Dockerfile容器化PySpark代码，目标是读取AWS S3公开桶中1.8MiB的Humorous.csv文件。当前运行代码时，打印完starting main function后就卡住了。我尝试配置Hadoop S3a参数但未解决问题，附上相关代码、日志和文件信息，恳请协助排查。

1. main.py代码 #

import os, datetime

from pyspark.sql.dataframe import DataFrame
from pyspark.sql.types import *
from pyspark.sql.functions import explode, split, col, sum, lit
from pyspark.sql import SparkSession

# os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages "org.apache.hadoop:hadoop-aws:3.3.1" pyspark-shell'


if __name__ == "__main__":
    print("starting main function")
    # build spark session
    spark = SparkSession.builder.appName("MainDemo").master("local[*]").getOrCreate()
    
    print(spark)
    print(type(spark))

    # Enable hadoop s3a settings
    # spark._jsc.hadoopConfiguration().set("com.amazonaws.services.s3.enableV4", "true")
    # spark._jsc.hadoopConfiguration().set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
    # spark._jsc.hadoopConfiguration().set("fs.AbstractFileSystem.s3a.impl", "org.apache.hadoop.fs.s3a.S3A")

    # read data from publc bucket into Spark DF
    data_path = "s3://humor-detection-pds/Humorous.csv" 
    df = spark.read.csv(data_path)

    # # apply spark transformations
    print(df.show())
    spark.stop()

2. 示例Dockerfile #

FROM bitnami/spark:3-debian-10
USER root
WORKDIR /opt/app
ADD requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
# USER 185
#Copy python script for batch
COPY main.py /opt/app/main.py

CMD ["/bin/bash"] # Ran the container specifying the entrypoint and tried running the code.

3. 更新后的运行日志 #

root@REDACTED_HOST:/opt/mlp# export AWS_SESSION_TOKEN="REDACTED_TOKEN";;
root@REDACTED_HOST:/opt/mlp# export AWS_SECRET_ACCESS_KEY="REDACTED_KEY";;
root@REDACTED_HOST:/opt/mlp# export AWS_ACCESS_KEY_ID="REDACTED_ID";;
root@REDACTED_HOST:/opt/mlp# cat <<EOF >>main.py
> import os, datetime
> from pyspark.sql.dataframe import DataFrame
> from pyspark.sql.types import *
> from pyspark.sql.functions import explode, split, col, sum, lit
> from pyspark.sql import SparkSession
>
> # os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages "org.apache.hadoop:hadoop-aws:3.3.1" pyspark-shell'
>
>
>if __name__ == "__main__":
>     print("starting main function")
>     # build spark session
>     spark = SparkSession.builder.appName("MainDemo").master("local[*]").getOrCreate()
>
>     print(spark)
>     print(type(spark))
>
>     # Enable hadoop s3a settings
>     spark._jsc.hadoopConfiguration().set("com.amazonaws.services.s3.enableV4", "true")
>     spark._jsc.hadoopConfiguration().set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
>     spark._jsc.hadoopConfiguration().set("fs.AbstractFileSystem.s3a.impl", "org.apache.hadoop.fs.s3a.S3A")
>
>     # read data from publc bucket into Spark DF
>     data_path = "s3a://humor-detection-pds/Humorous.csv"
>     df = spark.read.csv(data_path)
>
>     # # apply spark transformations
>     print(df.show()
> EOF
root@REDACTED_HOST:/opt/mlp# python main.py
starting main function
## Job is stuck here...

4. S3文件信息 #

$ aws s3 ls s3://humor-detection-pds/Humorous.csv --summarize --human-readable                                                          [ruby-2.6.8p205]
2020-05-20 04:41:39    1.8 MiB Humorous.csv

Total Objects: 1
   Total Size: 1.8 MiB

核心问题分析 #

代码卡住的核心原因：

1. 缺少S3访问依赖：PySpark默认未包含hadoop-aws及AWS Java SDK包，无法与S3建立有效通信
2. 配置错误：公开桶未启用匿名访问配置，且协议选择/参数设置不完整

修复步骤 #

1. 修改main.py，完善依赖与配置

取消注释依赖加载代码，补充S3a匿名访问等关键配置：

import os, datetime

from pyspark.sql.dataframe import DataFrame
from pyspark.sql.types import *
from pyspark.sql.functions import explode, split, col, sum, lit
from pyspark.sql import SparkSession

# 加载匹配版本的Hadoop AWS依赖包
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.hadoop:hadoop-aws:3.3.1,com.amazonaws:aws-java-sdk-bundle:1.12.262 pyspark-shell'


if __name__ == "__main__":
    print("starting main function")
    # 构建Spark会话
    spark = SparkSession.builder.appName("MainDemo").master("local[*]").getOrCreate()
    
    print(spark)
    print(type(spark))

    # 配置S3a参数
    hadoop_conf = spark._jsc.hadoopConfiguration()
    hadoop_conf.set("com.amazonaws.services.s3.enableV4", "true")
    hadoop_conf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
    hadoop_conf.set("fs.AbstractFileSystem.s3a.impl", "org.apache.hadoop.fs.s3a.S3A")
    # 公开桶启用匿名访问
    hadoop_conf.set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider")
    # 配置S3端点，避免DNS解析问题
    hadoop_conf.set("fs.s3a.endpoint", "s3.us-east-1.amazonaws.com")

    # 读取公开桶文件
    data_path = "s3a://humor-detection-pds/Humorous.csv" 
    df = spark.read.csv(data_path, header=True)  # 若文件含表头，添加header=True

    # 输出数据
    print(df.show())
    spark.stop()

2. 优化Dockerfile，简化运行流程

直接在镜像中配置依赖与运行命令，无需手动操作：

FROM bitnami/spark:3-debian-10
USER root
WORKDIR /opt/app

# 安装Python依赖（若requirements.txt有其他依赖）
ADD requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# 复制PySpark代码
COPY main.py /opt/app/main.py

# 提前设置依赖加载环境变量
ENV PYSPARK_SUBMIT_ARGS="--packages org.apache.hadoop:hadoop-aws:3.3.1,com.amazonaws:aws-java-sdk-bundle:1.12.262 pyspark-shell"

# 直接运行代码
CMD ["python", "/opt/app/main.py"]

3. 运行容器的正确命令

无需设置AWS密钥（公开桶无需认证），直接构建并运行：

# 构建镜像
docker build -t pyspark-s3-demo .

# 运行容器
docker run --rm pyspark-s3-demo

关键注意事项 #

• 依赖版本匹配：hadoop-aws版本必须与Spark内置Hadoop版本一致（Bitnami Spark 3.x对应Hadoop 3.3.1）
• 公开桶配置：必须启用AnonymousAWSCredentialsProvider，否则会尝试使用默认密钥导致卡住
• 协议选择：优先使用s3a://协议，s3://为旧实现，依赖包不同易出问题

内容的提问来源于stack exchange，提问作者bigtracker