寻求Apache Spark作业提交REST API最新文档及兼容问题解决方案

一、获取最新REST API文档的途径 #

• 官方文档中，Standalone模式的作业提交REST API说明位于对应版本文档的Spark Standalone章节下的REST API小节，直接查看你使用的Spark版本（如3.5.2）的官方文档即可找到完整的参数定义与取值说明。
• 若需更细致的参数细节，可查看Spark源码中org.apache.spark.deploy.rest包下的相关类（如CreateSubmissionRequest）的注释，里面包含所有请求参数的详细定义。

二、Spark 3.5.2作业提交示例 #

curl -X POST http://localhost:6066/v1/submissions/create \
--header "Content-Type:application/json;charset=UTF-8" \
--data '{
  "action":"CreateSubmissionRequest",
  "appArgs":[
    "/Users/tilak/jobs/test_job.py"
  ],
  "appResource":"file:/Users/tilak/jobs/test_job.py",
  "clientSparkVersion":"3.5.2",
  "environmentVariables":{
    "SPARK_ENV_LOADED":"1"
  },
  "mainClass":"org.apache.spark.deploy.SparkSubmit",
  "sparkProperties":{
    "spark.driver.supervise":"false",
    "spark.app.name":"Simple App",
    "spark.eventLog.enabled":"true",
    "spark.submit.deployMode":"cluster",
    "spark.master":"spark://localhost:6066"
  }
}'

三、Java 17兼容性错误解决 #

针对提交作业时出现的java.lang.IllegalAccessError错误，需在REST请求的sparkProperties中添加JVM模块访问权限配置，具体如下：

curl -X POST http://localhost:6066/v1/submissions/create \
--header "Content-Type:application/json;charset=UTF-8" \
--data '{
  "action":"CreateSubmissionRequest",
  "appArgs":[
    "/Users/tilak/jobs/test_job.py"
  ],
  "appResource":"file:/Users/tilak/jobs/test_job.py",
  "clientSparkVersion":"3.5.2",
  "environmentVariables":{
    "SPARK_ENV_LOADED":"1"
  },
  "mainClass":"org.apache.spark.deploy.SparkSubmit",
  "sparkProperties":{
    "spark.driver.supervise":"false",
    "spark.app.name":"Simple App",
    "spark.eventLog.enabled":"true",
    "spark.submit.deployMode":"cluster",
    "spark.master":"spark://localhost:6066",
    "spark.driver.extraJavaOptions": "--add-exports java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED",
    "spark.executor.extraJavaOptions": "--add-exports java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED"
  }
}'

上述配置中，spark.driver.extraJavaOptions和spark.executor.extraJavaOptions用于向Spark驱动和执行器的JVM传递参数，解除Java 17模块系统对内部API的访问限制，解决报错问题。

内容的提问来源于stack exchange，提问作者user27200545