通过Spark Kubernetes Operator提交Spring Boot Spark作业遇主类查找失败求助

问题背景 #

本地Standalone模式下，使用以下命令可正常提交Spring Boot Spark作业：

spark-submit --conf "spark.driver.userClassPathFirst=true" --conf "spark.executor.userClassPathFirst=true" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:///log4j2.xml -XX:ReservedCodeCacheSize=100M -XX:MaxMetaspaceSize=256m -XX:CompressedClassSpaceSize=256m --add-opens=java.base/sun.nio.ch=ALL-UNNAMED -Dlog4j.debug" --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=file:///log4j2.xml --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED -XX:ReservedCodeCacheSize=100M -XX:MaxMetaspaceSize=256m -XX:CompressedClassSpaceSize=256m -Dlog4j.debug" --driver-java-options "-Xms4096m -XX:+UseG1GC -XX:G1HeapRegionSize=32M --add-opens=java.base/sun.nio.ch=ALL-UNNAMED" --master spark://localhost:7077 --deploy-mode cluster --num-executors 1 --executor-cores 4 --executor-memory 4096m --driver-memory 4096m --conf "spark.driver.memory=4096m" --conf "spark.dynamicAllocation.enabled=true" operatordc1-0.0.1-SNAPSHOT.jar

在Kubernetes集群中部署ghcr.io/kubeflow/spark-operator:v1beta2-1.4.3-3.5.0后，编写SparkApplication YAML提交作业时出现错误。

提交的YAML配置 #

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: operatordc1
  namespace: spark-operator
spec:
  type: Java
  mode: cluster
  image: "focode/spark-custom:release-1.0"
  imagePullPolicy: Always
  mainApplicationFile: "local:///opt/spark/examples/jars/operatordc1-0.0.1-SNAPSHOT.jar"
  sparkVersion: "3.4.2"
  restartPolicy:
    type: Never
  driver:
    cores: 1
    coreLimit: "1000m"
    memory: "1024m"
    javaOptions: >-
      -Dlog4j.configuration=file:///log4j2.xml
      --add-opens=java.base/java.lang=ALL-UNNAMED
      --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
      --add-opens=java.base/java.nio=ALL-UNNAMED
      --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
      --add-opens=java.base/java.util=ALL-UNNAMED
      --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
      --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED
      -XX:+UseG1GC
      -XX:G1HeapRegionSize=32M
      -XX:ReservedCodeCacheSize=100M
      -XX:MaxMetaspaceSize=256m
      -XX:CompressedClassSpaceSize=256m
      -Xms1024m
      -Dlog4j.debug
    labels:
      version: "3.4.2"
    serviceAccount: default
  executor:
    cores: 4
    instances: 1
    memory: "1024m"
    javaOptions: >-
      -Dlog4j.configuration=file:///log4j2.xml
      -XX:ReservedCodeCacheSize=100M
      -XX:MaxMetaspaceSize=256m
      -XX:CompressedClassSpaceSize=256m
      --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
      -Dlog4j.debug
    labels:
      version: "3.4.2"
    serviceAccount: default
  sparkConf:
    "spark.driver.userClassPathFirst": "true"
    "spark.executor.userClassPathFirst": "true"
    "spark.driver.memory": "1024m"
    "spark.executor.memory": "1024m"
    "spark.dynamicAllocation.enabled": "true"

错误信息 #

failed to submit SparkApplication operatordc1: failed to run spark-submit for SparkApplication spark-operator/operatordc1: 24/04/27 10:19:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" org.apache.spark.SparkException: Failed to get main class in JAR with error 'No FileSystem for scheme "local"'. Please specify one with --class. at org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:1047) at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:528) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:964) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

解决方案 #

1. 修正文件路径协议 #

错误核心是No FileSystem for scheme "local"，Spark在Kubernetes环境中对local://协议处理存在兼容性问题，将mainApplicationFile协议改为file://：

mainApplicationFile: "file:///opt/spark/examples/jars/operatordc1-0.0.1-SNAPSHOT.jar"

2. 指定Spring Boot可执行Jar的正确主类 #

Spring Boot可执行Jar的启动主类并非业务代码主类，而是org.springframework.boot.loader.JarLauncher，需在YAML中配置：

mainClass: "org.springframework.boot.loader.JarLauncher"

（注：Spring Boot 1.x版本主类为org.springframework.boot.loader.JarLauncher或org.springframework.boot.loader.WarLauncher，根据打包类型确定）

3. 验证Jar包在镜像中的存在性 #

确保作业Jar包存在于自定义Spark镜像的指定路径，可通过以下命令验证：

docker run --rm focode/spark-custom:release-1.0 ls /opt/spark/examples/jars/

4. 对齐Spark版本 #

Spark Operator使用3.5.0版本，YAML中sparkVersion为3.4.2，版本差异可能导致兼容性问题，建议调整为3.5.0：

sparkVersion: "3.5.0"

修改后的YAML核心片段 #

spec:
  type: Java
  mode: cluster
  image: "focode/spark-custom:release-1.0"
  imagePullPolicy: Always
  mainApplicationFile: "file:///opt/spark/examples/jars/operatordc1-0.0.1-SNAPSHOT.jar"
  mainClass: "org.springframework.boot.loader.JarLauncher"
  sparkVersion: "3.5.0"
  # 其他配置保持不变

验证提交 #

修改完成后重新提交作业：

kubectl apply -f spark-job-poc-1.yaml -n spark-operator

内容的提问来源于stack exchange，提问作者focode