Spark提交任务遇ClassNotFoundException：org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException

问题场景 #

搭建1主2从的3节点Hadoop（Yarn）+Spark集群，PySpark脚本依赖org.elasticsearch.spark写入Elasticsearch，执行spark-submit时通过--packages org.elasticsearch:elasticsearch-spark-30_2.12:8.4.1引入依赖，触发如下错误：

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/javax/ws/rs/core/NoContentException
        at org.apache.hadoop.yarn.util.timeline.TimelineUtils.<clinit>(TimelineUtils.java:60)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:200)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
        at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:191)
        at org.apache.spark.deploy.yarn.Client.run(Client.scala:1327)
        at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1764)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 13 more

已尝试无效方案 #

• 添加Stack Overflow指定的所有路径
• 将Hadoop从3.1.1升级到3.3.3
• 手动下载对应jar包到spark/jars目录，不带--packages执行spark-submit

解决方案 #

1. 排除冲突依赖并启用用户类路径优先 #

该错误本质是Elasticsearch Spark依赖引入的非shaded版javax.ws.rs与Hadoop自带的shaded版冲突，导致JVM找不到Hadoop的NoContentException类。执行spark-submit时添加类路径优先级配置：

spark-submit \
  --packages org.elasticsearch:elasticsearch-spark-30_2.12:8.4.1 \
  --conf spark.driver.userClassPathFirst=true \
  --conf spark.executor.userClassPathFirst=true \
  your_pyspark_script.py

userClassPathFirst=true会让Spark优先加载用户指定的依赖，避免Hadoop的shaded类被覆盖。

2. 手动引入Hadoop Yarn核心包 #

org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException包含在Hadoop的hadoop-yarn-common包中，手动将对应Hadoop版本的该包加入依赖：

spark-submit \
  --packages org.elasticsearch:elasticsearch-spark-30_2.12:8.4.1 \
  --jars $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-common-3.3.3.jar \
  your_pyspark_script.py

请替换3.3.3为你的Hadoop实际版本。

3. 修改Spark全局类路径配置 #

在spark-env.sh中添加Hadoop Yarn核心包到Spark的类路径：

echo 'export SPARK_CLASSPATH=$SPARK_CLASSPATH:$HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-common-3.3.3.jar' >> $SPARK_HOME/conf/spark-env.sh

重启Spark服务后再执行spark-submit，无需每次手动指定jar包。

4. 验证版本兼容性 #

确认elasticsearch-spark-30_2.12:8.4.1与Hadoop 3.3.3的兼容性，若存在适配问题，更换为Elasticsearch官方文档中推荐的对应Hadoop版本的Spark连接器。

内容的提问来源于stack exchange，提问作者Sowjanya R Bhat