Spark3.3.0 on Kubernetes提交PySpark任务触发Ivy异常咨询

需要在Kubernetes集群上运行无业务逻辑的极简PySpark测试程序，仅用于验证基础环境可用性，待环境验证通过后再扩展实际业务逻辑。测试程序simpleapp.py代码如下：

from pyspark.sql import SparkSession

print("simple pyspark app starting.")

spark = SparkSession.builder.appName("SimpleApp").getOrCreate()
print("created spark session")

spark.stop()
print("done")

使用本地Spark 3.3.0发行版自带的spark-submit工具提交任务，提交命令如下：

~/opt/spark/current/bin/spark-submit \
      --master k8s://<redacted>:443 \
      --deploy-mode cluster \
      --name pyspark-test \
      --packages "org.apache.hadoop:hadoop-aws:3.3.2" \
      --conf spark.executor.instances=1 \
      --conf "spark.kubernetes.container.image=apache/spark-py:v3.3.0" \
      --conf "spark.kubernetes.file.upload.path=s3a://pyspark-test" \
      --conf "spark.hadoop.fs.s3a.access.key=<redacted>" \
      --conf "spark.hadoop.fs.s3a.secret.key=<redacted>" \
      --conf spark.kubernetes.namespace=pysparktest \
      ./simpleapp.py

提交后观察到simpleapp.py已成功上传至S3存储，Kubernetes集群pysparktest命名空间下也成功启动对应Pod，但任务运行失败。查看Pod日志得到报错如下：

kubectl -n pysparktest logs --tail=200 -lspark-app-name=pyspark-test

++ id -u
+ myuid=185
++ id -g
+ mygid=0
+ set +e
++ getent passwd 185
+ uidentry=
+ set -e
+ '[' -z '' ']'
+ '[' -w /etc/passwd ']'
+ echo '185:x:185:0:anonymous uid:/opt/spark:/bin/false'
+ '[' -z /usr/local/openjdk-11 ']'
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sort -t_ -k4 -n
+ sed 's/[^=]*=\(.*\)/\1/g'
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -z ']'
+ '[' -z ']'
+ '[' -n '' ']'
+ '[' -z ']'
+ '[' -z x ']'
+ SPARK_CLASSPATH='/opt/spark/conf::/opt/spark/jars/*'
+ case "$1" in
+ shift 1
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@")
+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.124.35.246 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.PythonRunner s3a://pyspark-test/spark-upload-4c10a3df-14c5-46a9-a940-30548a7f586f/simpleapp.py
:: loading settings :: url = jar:file:/opt/spark/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /opt/spark/.ivy2/cache
The jars for the packages stored in: /opt/spark/.ivy2/jars
org.apache.hadoop#hadoop-aws added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-29bd23dc-1c5d-4550-ac2f-47ddbeb45f8e;1.0
    confs: [default]
Exception in thread "main" java.io.FileNotFoundException: /opt/spark/.ivy2/cache/resolved-org.apache.spark-spark-submit-parent-29bd23dc-1c5d-4550-ac2f-47ddbeb45f8e-1.0.xml (No such file or directory)
    at java.base/java.io.FileOutputStream.open0(Native Method)
    at java.base/java.io.FileOutputStream.open(Unknown Source)
    at java.base/java.io.FileOutputStream.<init>(Unknown Source)
    at java.base/java.io.FileOutputStream.<init>(Unknown Source)
    at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorWriter.write(XmlModuleDescriptorWriter.java:71)
    at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorWriter.write(XmlModuleDescriptorWriter.java:63)
    at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.toIvyFile(DefaultModuleDescriptor.java:553)
    at org.apache.ivy.core.cache.DefaultResolutionCacheManager.saveResolvedModuleDescriptor(DefaultResolutionCacheManager.java:183)
    at org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:259)
    at org.apache.ivy.Ivy.resolve(Ivy.java:522)
    at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1454)
    at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:185)
    at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:901)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

存在两点疑问：

• Ivy是Java生态的依赖管理工具，提交的只是极简PySpark测试脚本，为何会触发Ivy相关异常？
• 是否有更简便的PySpark on Kubernetes运行方案，是否需要放弃官方apache/spark-py:v3.3.0镜像，转而构建内置simpleapp.py的自定义镜像运行任务？

这个Ivy报错和PySpark脚本本身没有关系，触发源是提交命令中的--packages "org.apache.hadoop:hadoop-aws:3.3.2"参数：

• 只要提交任务时携带--packages参数，无论任务是Java/Scala还是Python编写，Spark都会在启动阶段调用内置的Ivy依赖管理工具，从公共Maven仓库拉取指定的第三方依赖包，写入本地Ivy缓存目录。
• 官方apache/spark-py:v3.3.0镜像默认以UID 185的非root用户启动，镜像内默认的Ivy缓存路径/opt/spark/.ivy2归属root用户，且未对普通用户开放写权限，Ivy写入缓存文件时直接触发权限不足的文件找不到错误。

最快验证方案 #

当前测试脚本没有任何S3读写逻辑，根本不需要hadoop-aws依赖，直接把提交命令里的--packages "org.apache.hadoop:hadoop-aws:3.3.2"参数删掉，就不会触发Ivy依赖拉取流程，空测试脚本可以直接跑通。

临时测试可用方案 #

如果确实需要在测试阶段动态加载依赖，不需要修改基础镜像，只需要在提交命令中增加一条配置，把Ivy缓存路径指定到普通用户有写权限的临时目录即可：

--conf spark.jars.ivy=/tmp/.ivy

这个方案的缺点是每次启动Driver和Executor Pod都要重新拉取依赖，冷启动慢，网络波动时容易拉取失败，只适合临时调试使用。

生产环境推荐方案 #

不要依赖运行时动态拉取依赖，基于官方Spark镜像构建自定义镜像，提前把需要的第三方依赖、业务代码都打包进镜像内：

1. 编写Dockerfile：

FROM apache/spark-py:v3.3.0
USER root
# 提前下载hadoop-aws及配套依赖到Spark的jars目录
RUN cd /opt/spark/jars && \
    curl -O https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.2/hadoop-aws-3.3.2.jar && \
    curl -O https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.1026/aws-java-sdk-bundle-1.11.1026.jar
# 可直接将业务PySpark脚本拷贝到镜像工作目录
COPY simpleapp.py /opt/spark/work-dir/
USER 185

2. 构建镜像并推送到K8s集群可访问的镜像仓库，提交任务时将spark.kubernetes.container.image配置改为自定义镜像地址，去掉--packages参数即可。

• 这种方案不存在运行时拉依赖的不稳定问题，任务启动速度快，是PySpark on K8s生产部署的标准实践。
• 如果业务代码经常变动，不需要每次把代码打进镜像，依然可以用S3上传的方式分发代码，只要把依赖提前打进镜像就行。

内容的提问来源于stack exchange，提问作者clay