在Minikube集群中使用Petastorm v0.11.4运行Spark任务时Executor容器启动失败问题求助

I've run into similar issues when setting up Spark on Kubernetes with custom images, let me break down what's going on and how to fix it:

The Root Cause #

The error exec: "executor": executable file not found in $PATH means Kubernetes is trying to start your executor container with the command executor, but this binary doesn't exist in your custom petastorm:v0.0.1 image, or isn't in the system's PATH.

Spark on Kubernetes expects the executor container to have a valid entry point that can bootstrap the Spark executor process. If you built your image from scratch (not based on an official Spark image), it likely lacks the necessary Spark binaries and configuration to handle this.

Fix 1: Use an Official Spark Image as Base for Your Custom Image #

The easiest way to resolve this is to build your Petastorm-enabled image on top of an official Spark image, which already includes all the required Spark binaries and executor entry points. Here's an example Dockerfile:

# Use a compatible Spark version (match what you're running locally)
FROM apache/spark:3.3.0
USER root

# Install Petastorm v0.11.4 and any other dependencies
RUN pip install petastorm==0.11.4

# Switch back to the default spark user for security
USER spark

Rebuild your image and push it to your Minikube registry, then update your Spark config to use this new image.

Fix 2: Manually Configure Executor Command and Args #

If you need to keep your existing base image, you can explicitly tell Spark what command to run to start the executor. Add these lines to your SparkConf setup:

# Point to the spark-class executable (adjust path to match your Spark installation in the image)
spark_conf.set("spark.kubernetes.executor.command", "/opt/spark/bin/spark-class")
# Pass the required arguments to bootstrap the executor
spark_conf.set("spark.kubernetes.executor.args", 
    "org.apache.spark.executor.CoarseGrainedExecutorBackend "
    "--driver-url $SPARK_DRIVER_URL "
    "--executor-id $SPARK_EXECUTOR_ID "
    "--cores $SPARK_EXECUTOR_CORES "
    "--app-id $SPARK_APPLICATION_ID "
    "--hostname $SPARK_EXECUTOR_POD_NAME"
)

Make sure the path /opt/spark/bin/spark-class matches where Spark is installed in your image. The environment variables like $SPARK_DRIVER_URL are automatically injected by Spark when launching the executor pods.

Fix 3: Verify PATH in Your Image #

If you prefer to use the default executor command, you can create a wrapper script named executor in a directory that's in your image's PATH. For example, create a script at /usr/local/bin/executor with:

#!/bin/bash
/opt/spark/bin/spark-class org.apache.spark.executor.CoarseGrainedExecutorBackend "$@"

Then make it executable in your Dockerfile:

RUN chmod +x /usr/local/bin/executor

Additional Checks #

• Double-check that your driver host configuration (spark.driver.host) matches the actual name of your driver pod. You can also use the environment variable SPARK_DRIVER_POD_NAME to auto-populate this instead of hardcoding it:

  import os
  spark_conf.set("spark.driver.host", os.environ.get("SPARK_DRIVER_POD_NAME"))
  

• Ensure your Minikube registry is properly accessible and your image was pushed correctly. You can verify with minikube image ls to confirm the image exists locally.

内容的提问来源于stack exchange，提问作者skynet1010