EMR 6.4.0运行PySpark应用遇两类错误求助

一、类未找到异常（NoClassDefFoundError） #

报错信息 #

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/javax/ws/rs/core/NoContentException
    at org.apache.hadoop.yarn.util.timeline.TimelineUtils.<clinit>(TimelineUtils.java:60)
    at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:200)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
    at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:191)
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1327)
    at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1764)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 13 more
Command exiting with ret '1'

解决方法 #

• 匹配Snowflake connector版本：EMR 6.4.0默认搭载Spark 3.2.1，你当前使用的spark-snowflake_2.12-2.9.1-spark_3.1.jar是Spark 3.1适配版，版本不匹配会引发依赖冲突。更换为适配Spark 3.2的版本（如spark-snowflake_2.12-2.10.0-spark_3.2.jar及以上）。
• 添加Hadoop核心依赖到driver类路径：在spark-submit命令中增加配置：--conf spark.driver.extraClassPath=/usr/lib/hadoop/hadoop-common.jar，该Jar包含缺失的org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException类。
• 排除冲突依赖：如果自定义Jar包中包含未shaded的javax.ws.rs相关类，会和Hadoop的shaded版本冲突。打包时可排除这类依赖，或临时开启--conf spark.driver.userClassPathFirst=true让用户类路径优先（注意该配置可能引发其他兼容性问题，需测试）。

二、YARN资源请求错误 #

提交命令 #

spark-submit --deploy-mode cluster --master yarn --driver-memory 4g --executor-memory 1g --executor-cores 1 --num-executors 1  --conf spark.rpc.message.maxSize=100  --jars /home/hadoop/configure_cluster/snowflake-jdbc-3.13.8.jar,/home/hadoop/configure_cluster/spark-snowflake_2.12-2.9.1-spark_3.1.jar --py-files /home/hadoop/spark_utils.zip /home/hadoop/weibull_2.py dev dafehv-dse-weibull-processing-dev

报错信息 #

diagnostics: Uncaught exception: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request! Cannot allocate containers as requested resource is greater than maximum allowed allocation. Requested resource type=[memory-mb], Requested resource=<memory:35789, max memory:2147483647, vCores:2, max vCores:2147483647>, maximum allowed allocation=<memory:6144, vCores:4>, please note that maximum allowed allocation is calculated by scheduler based on maximum resource of registered NodeManagers, which might be less than configured maximum allocation=<memory:6144, vCores:128>

解决方法 #

• 排查代码中的资源配置覆盖：报错显示请求内存达35789MB（约35GB），远超出你指定的4GB driver内存。检查PySpark代码中是否有spark.conf.set("spark.driver.memory", ...)这类语句，是否不小心设置了错误的内存值（比如单位写错、数值输入错误）。
• 调整driver内存至YARN允许范围：M4.large实例总内存为8GB，EMR默认给YARN容器的最大分配内存是6144MB（6GB）。尝试将driver内存调低至3GB（--driver-memory 3g），避免接近上限引发的预留内存计算问题。
• 检查YARN核心配置：登录EMR主节点，查看/etc/hadoop/conf/yarn-site.xml文件，确认yarn.nodemanager.resource.memory-mb和yarn.scheduler.maximum-allocation-mb的值是否为6144MB。若需要更大容器，可通过EMR集群配置修改参数，但受M4.large硬件限制，不建议超过6GB。
• 修正vCores配置：报错中请求vCores为2，但你指定的--executor-cores 1，cluster模式下driver（即YARN AM）默认vCores为1。检查代码中是否设置了spark.driver.cores=2，若不需要，可在提交命令中添加--conf spark.driver.cores=1。

内容的提问来源于stack exchange，提问作者Akshay