Spark on Kubernetes执行Python脚本报错：找不到python3

问题重现 #

构建镜像命令 #

./bin/docker-image-tool.sh -r pyspark -t 3.3.1-2 -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile build

提交K8s任务命令 #

./bin/spark-submit --master k8s://https://172.16.0.6:6443 --deploy-mode cluster \
    --conf spark.kubernetes.namespace=techcool \
    --name spark-pi \
    --conf spark.executor.instances=5 \
    --conf spark.kubernetes.container.image=pyspark/spark:3.3.1-2 \
    local:///opt/spark/examples/src/main/python/pi.py

报错信息 #

+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.244.36.87 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.PythonRunner local:///opt/spark/examples/src/main/python/pi.py

23/05/24 09:02:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Exception in thread "main" java.io.IOException: Cannot run program "python3": error=2, No such file or directory

已验证项 #

• 本地运行镜像可正常执行脚本：

docker run -it --rm pyspark/spark-py:3.3.1-2 driver --class org.apache.spark.deploy.PythonRunner local:///opt/spark/examples/src/main/python/pi.py

• 尝试指定K8s环境变量未解决问题：

--conf spark.kubernetes.driverEnv.PYSPARK_DRIVER_PYTHON=/usr/bin/python3 \
--conf spark.executorEnv.PYSPARK_PYTHON=/usr/bin/python3

解决方案 #

方案1：修正Spark Python解释器配置 #

在spark-submit命令中直接指定Spark级别的Python解释器路径，而非仅通过K8s环境变量传递：

./bin/spark-submit --master k8s://https://172.16.0.6:6443 --deploy-mode cluster \
    --conf spark.kubernetes.namespace=techcool \
    --name spark-pi \
    --conf spark.executor.instances=5 \
    --conf spark.kubernetes.container.image=pyspark/spark:3.3.1-2 \
    --conf spark.pyspark.python=/usr/bin/python3 \
    --conf spark.pyspark.driver.python=/usr/bin/python3 \
    local:///opt/spark/examples/src/main/python/pi.py

方案2：确保镜像中Python路径在PATH环境变量内 #

1. 进入构建好的镜像检查PATH：

docker run -it --rm pyspark/spark:3.3.1-2 echo $PATH

2. 如果/usr/bin不在PATH中，修改Dockerfile添加：

ENV PATH="/usr/bin:$PATH"

3. 重新构建镜像并提交任务。

方案3：检查Dockerfile的Python安装逻辑 #

确认Dockerfile中正确安装了python3且路径正确，例如：

RUN apt-get update && apt-get install -y --no-install-recommends python3 && \
    rm -rf /var/lib/apt/lists/*

内容的提问来源于stack exchange，提问作者两极化