Spark Submit执行Shaded Jar遇ClassNotFoundException问题解决

问题场景 #

通过Spark Standalone集群的REST API提交shaded jar执行Beam任务时，Executor抛出异常：

ClassNotFoundException: org.apache.beam.runners.spark.io.SourceRDD$SourcePartition

提交请求的curl命令如下：

curl -X POST http://xxxxxx:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
  "appResource": "/home/xxxx/xxxx-bundled-0.1.jar",
  "sparkProperties": {
    "spark.master": "spark://xxxxxxx:7077",
    "spark.driver.userClassPathFirst": "true",
    "spark.executor.userClassPathFirst": "true",
    "spark.app.name": "DataPipeline",
    "spark.submit.deployMode": "cluster",
    "spark.driver.supervise": "true"
  },
  "environmentVariables": {
    "SPARK_ENV_LOADED": "1"
  },
  "clientSparkVersion": "3.1.3",
  "mainClass": "com.xxxx.DataPipeline",
  "action": "CreateSubmissionRequest",
  "appArgs": [
    "--config=xxxx",
    "--runner=SparkRunner"
  ]
}'

环境信息 #

• Beam版本：2.41.0
• Spark版本：3.1.3
• Hadoop版本：3.2.0
• 项目依赖proto3，因此最初设置了spark.driver.userClassPathFirst和spark.executor.userClassPathFirst为true

解决方法 #

1. 将shaded插件升级至3.4.0版本
2. 移除Spark配置中的spark.driver.userClassPathFirst和spark.executor.userClassPathFirst两项配置

调整后，本地运行及通过REST API提交任务均恢复正常。

内容的提问来源于stack exchange，提问作者user20420926