Spark on EKS：S3AFileSystem类在Driver可用但Executor报错缺失

环境背景 #

• 使用Spark 3.3.4的docker-image-tool.sh工具构建自定义PySpark镜像，已预装hadoop-aws和aws-java-sdk-bundle jar包
• 部署在EKS 1.29集群，使用最新版spark-operator，构建operator镜像时指定该自定义镜像为SPARK_IMAGE参数
• SparkApplication使用s3a://路径存储主应用文件、pyFiles、files，Driver可正常拉取这些资源并执行部分S3操作，但Executor报错

错误信息 #

Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2686) 

SparkApplication配置 #

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: spark-${RUN_ID}
  namespace: spark
spec:
  type: Python
  #pythonVersion: "3"
  mode: cluster
  image: "mjans71/spark-py:3.3.4"
  imagePullPolicy: Always
  mainApplicationFile: "s3a://${S3_BUCKET}/app/app.py"
  sparkVersion: "3.3.4"
  restartPolicy:
    type: Never
  deps:
    packages:
      - "org.apache.hadoop:hadoop-aws:3.3.2"
    pyFiles:
      - "s3a://${S3_BUCKET}/app/pyfiles.zip"
    files:
      - "s3a://${S3_BUCKET}/app/files.tar.gz"
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 3.3.4
    serviceAccount: spark-driver
    env:
      - name: STORAGE_PATH
        value: "s3a://${S3_BUCKET}/data"
      - name: RESULTS_PATH
        value: "s3a://${S3_BUCKET}/results"
      - name: RUN_ID
        value: "${RUN_ID}"
      - name: QUERY_GLOB
        value: "*"
  executor:
    cores: 4
    instances: 4
    serviceAccount: spark-driver
    memory: "512m"
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: spark-role
              operator: In
              values:
              - executor
          topologyKey: "kubernetes.io/hostname"
  sparkConf:
    "spark.driver.extraJavaOptions": "-Divy.cache.dir=/tmp -Divy.home=/tmp"
    "spark.executor.extraJavaOptions": "-Divy.cache.dir=/tmp -Divy.home=/tmp"

spark-operator执行的spark-submit命令 #

/opt/spark/bin/spark-submit \
    --master k8s://https://10.100.0.1:443 \
    --deploy-mode cluster \
    --conf spark.kubernetes.namespace=spark \
    --conf spark.app.name=spark-1 \
    --conf spark.kubernetes.driver.pod.name=spark-1-driver \
    --files s3a://bucketname/app/queries.tar.gz \
    --py-files s3a://bucketname/app/pyfiles.zip \
    --packages org.apache.hadoop:hadoop-aws:3.3.2 \
    --conf spark.kubernetes.container.image=mjans71/spark-py:3.3.4 \
    --conf spark.kubernetes.container.image.pullPolicy=Always \
    --conf spark.kubernetes.submission.waitAppCompletion=false \
    --conf spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp \
    --conf spark.executor.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp \
    --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/app-name=spark-1 \
    --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/launched-by-spark-operator=true \
    --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/submission-id=b0d78417-6a5d-4be4-b056-c18713bdbbc6 \
    --conf spark.driver.cores=1 \
    --conf spark.kubernetes.driver.limit.cores=1200m \
    --conf spark.driver.memory=512m \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-driver \
    --conf spark.kubernetes.driver.label.version=3.3.4 \
    --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/app-name=spark-1 \
    --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/launched-by-spark-operator=true \
    --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/submission-id=b0d78417-6a5d-4be4-b056-c18713bdbbc6 \
    --conf spark.executor.instances=4 \
    --conf spark.executor.cores=4 \
    --conf spark.executor.memory=512m \
    s3a://bucketname/app/run-queries.py

补充信息 #

• 设置spark.kubernetes.executor.deleteOnTermination后获取到Executor日志，未发现Executor从Driver下载hadoop-aws包的记录
• 尝试将hadoop-aws相关jar包放到镜像的/opt/spark/jars和/opt/spark/work-dir目录，无论是否保留--packages参数，均出现相同报错

核心疑问 #

spec.packages/--packages参数应该能确保Driver和Executor都获取到必要依赖，为什么Executor还是报找不到S3AFileSystem类？

内容的提问来源于stack exchange，提问作者MikeJansen