在Kubernetes中使用Spark Operator运行PySpark任务失败求助

本地已部署minikube集群、Spark Operator，且配置了所需的ServiceAccount与RBAC。提交SparkApplication运行PySpark应用时失败，报错如下：

Error: failed to start container "spark-kubernetes-driver": Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "driver": executable file not found in $PATH: unknown

将该应用作为K8s Job运行可成功执行，单独运行镜像也正常，说明应用代码无问题。以下是使用的Dockerfile和SparkApplication清单：

Dockerfile #

# Use an official OpenJDK runtime as a parent image
FROM openjdk:8-jdk

# Set environment variables
ENV PYSPARK_PYTHON=python3
ENV PYTHON_VERSION=3.8.12
ENV SPARK_VERSION=3.2.0
ENV HADOOP_VERSION=3.2

# Install Python and necessary dependencies
RUN apt-get update && \
    apt-get install -y python3 python3-pip && \
    apt-get clean

# Install PySpark
RUN pip3 install pyspark==$SPARK_VERSION

# Set up environment variables for Spark
ENV SPARK_HOME=/spark
ENV PATH=$PATH:$SPARK_HOME/bin
ENV PYTHONPATH=$SPARK_HOME/python:$PYTHONPATH

# Download and extract Apache Spark
RUN wget -qO- https://archive.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop$HADOOP_VERSION.tgz | tar -xvz -C / && \
    mv /spark-$SPARK_VERSION-bin-hadoop$HADOOP_VERSION /spark


# Set working directory
WORKDIR /app

# Copy your application files to the container
COPY . .

RUN mkdir /app/jars

# Set dependencies
RUN wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-azure/3.3.1/hadoop-azure-3.3.1.jar -P /app/jars

RUN chmod 775 ./src/python/spark_app.py

# Define the command to run your application
ENTRYPOINT ["python3","./src/python/spark_app.py"]

SparkApplication清单 #

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: spark-apps
spec:
  type: Python
  mode: cluster
  image: "image:tag"
  imagePullPolicy: IfNotPresent 
  mainApplicationFile: "local:///app/src/python/spark_app.py"
  sparkVersion: "3.2.0"
  restartPolicy:
    type: Never
  driver:
    cores: 1
    memory: "512m"
    labels:
      version: 3.2.0
    serviceAccount: spark
  executor:
    cores: 1
    instances: 1
    memory: "512m"
    labels:
      version: 3.2.0

问题根源在于Dockerfile的ENTRYPOINT设置。Spark Operator在集群模式下运行PySpark应用时，会向容器传递driver命令启动Spark驱动进程，但你的镜像设置了固定ENTRYPOINT直接运行Python脚本，覆盖了Operator的启动命令，导致容器找不到driver可执行文件。

有两种解决方式：

• 方式一：移除Dockerfile中的ENTRYPOINT
  删除Dockerfile最后一行的ENTRYPOINT ["python3","./src/python/spark_app.py"]。Spark Operator会通过mainApplicationFile指定的路径启动应用，无需在镜像中预设启动命令。

• 方式二：将ENTRYPOINT改为CMD
  如果需要保留镜像单独运行的能力，可将ENTRYPOINT替换为CMD：

  CMD ["python3","./src/python/spark_app.py"]
  

  这样单独运行镜像时会执行Python脚本，而Spark Operator启动时能传递自己的driver命令覆盖默认行为。

修改后重新构建镜像，提交SparkApplication即可正常运行。

内容的提问来源于stack exchange，提问作者Vaibhav