Minikube上Spark应用Executor实例无法创建的问题排查

在Minikube上使用spark-on-k8s-operator运行Java Spark应用时，Driver Pod启动成功，但Executor Pod未创建，日志显示Executor在Driver Pod内运行，Spark UI显示spark.master local[*]，说明应用以本地模式运行，未使用K8S集群模式。

相关代码片段：

static void minIOReadTester2()
{
     System.out.println("*************** Getting spark session ****************");
     SparkSession spark = SparkSession.builder()
            .appName("spark-with-nats-readMinIO")
            .config("fs.s3a.access.key", "minioadmin")
            .config("fs.s3a.secret.key", "minioadmin")
            .config("fs.s3a.endpoint", "http://minio-service.minio-dev:9000")
            .config("fs.s3a.connection.ssl.enabled", "true")
            .config("fs.s3a.path.style.access", "true")
            .config("fs.s3a.attempts.maximum", "1")
            .config("fs.s3a.connection.establish.timeout", "5000")
            .config("fs.s3a.connection.timeout", "10000")
            .getOrCreate();
     System.out.println("*************** Got spark session ****************");
     Dataset<Row> dftest = spark.read().load("s3a://testbucket/outputdelta");
     System.out.println("*** COUNT *** : "+dftest.count());
     dftest.show(20,false);
}

SparkApplication CRD配置文件：

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-app
  namespace: minio-dev
  labels:
    app: spark-app
spec:
  type: Java
  mode: cluster  # Deploy in cluster mode
  image: dockerhub/spark-minio:latest
  imagePullPolicy: Always
  mainClass: com.test.MinIOTester
  mainApplicationFile: local:///app/spark-learning-0.0.1-SNAPSHOT.jar
  sparkVersion: 3.3.3
  deps:
    jars:
     - local:///app/libs/nats-spark-connector-balanced_2.12-1.1.4.jar
     - local:///app/libs/jnats-2.17.1.jar
  driver:
    cores: 1
    memory: 1024m
    serviceAccount: sparksvcnew
    labels:
      version: 3.3.3
  executor:
    cores: 1
    instances: 2
    memory: 1024m
    serviceAccount: sparksvcnew
    labels:
      version: 3.3.3
  imagePullSecrets:
  - dockerhub-secret

关键日志片段：

24/02/01 13:51:46 INFO Executor: Starting executor ID driver on host spark-app-driver
24/02/01 13:51:46 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''

K8s事件输出：

7m41s       Normal    SparkApplicationSubmitted             sparkapplication/spark-app   SparkApplication spark-app was submitted successfully
7m34s       Normal    SparkDriverRunning                    sparkapplication/spark-app   Driver spark-app-driver is running
7m16s       Normal    SparkDriverCompleted                  sparkapplication/spark-app   Driver spark-app-driver completed
7m16s       Normal    SparkApplicationCompleted             sparkapplication/spark-app   SparkApplication spark-app completed

核心问题是spark.master被设置为local[*]，导致应用以本地模式运行，不会创建独立的Executor Pod，触发原因包括：

• 代码中构建SparkSession时，未继承operator传递的K8s集群配置，而是从头构建，Spark默认使用本地模式。
• 所使用的Spark镜像中spark-defaults.conf配置了spark.master=local[*]，覆盖了operator自动注入的K8s集群master地址。
• SparkApplication CRD未显式指定spark.master，且代码未加载系统默认的SparkConf，导致operator的配置未生效。

方法一：在CRD中显式指定spark.master #

在SparkApplication CRD的spec中添加sparkConf配置，明确指定K8s集群的master地址：

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-app
  namespace: minio-dev
spec:
  # 保留原有配置...
  sparkConf:
    spark.master: "k8s://https://kubernetes.default.svc"

方法二：修改代码继承系统配置 #

修改Java代码，先加载系统传递的SparkConf，再添加自定义配置，确保继承operator的集群模式设置：

static void minIOReadTester2()
{
     System.out.println("*************** Getting spark session ****************");
     // 先加载系统默认的SparkConf，包含operator传递的master配置
     SparkConf conf = new SparkConf();
     SparkSession spark = SparkSession.builder()
            .config(conf)
            .appName("spark-with-nats-readMinIO")
            .config("fs.s3a.access.key", "minioadmin")
            .config("fs.s3a.secret.key", "minioadmin")
            .config("fs.s3a.endpoint", "http://minio-service.minio-dev:9000")
            .config("fs.s3a.connection.ssl.enabled", "true")
            .config("fs.s3a.path.style.access", "true")
            .config("fs.s3a.attempts.maximum", "1")
            .config("fs.s3a.connection.establish.timeout", "5000")
            .config("fs.s3a.connection.timeout", "10000")
            .getOrCreate();
     System.out.println("*************** Got spark session ****************");
     Dataset<Row> dftest = spark.read().load("s3a://testbucket/outputdelta");
     System.out.println("*** COUNT *** : "+dftest.count());
     dftest.show(20,false);
}

方法三：检查并修改Spark镜像配置 #

检查所使用的dockerhub/spark-minio:latest镜像中的spark-defaults.conf文件，删除spark.master=local[*]这类强制设置本地模式的配置项。

内容的提问来源于stack exchange，提问作者VGH