EMR运行Spark Python应用遇SparkException，求排查方案

问题场景 #

• 操作：在EMR上运行Spark官方教程的圆周率计算Python应用，将.py文件上传至S3后，通过添加集群步骤启动任务，但误将.py文件指定为JAR文件
• 报错日志：

22/12/20 00:52:24 ERROR ApplicationMaster: Uncaught exception: 
org.apache.spark.SparkException: Exception thrown in awaitResult: 
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0]
    at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:514) ~[spark-yarn_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0]
    at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:278) ~[spark-yarn_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0]
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:929) ~[spark-yarn_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0]
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:928) ~[spark-yarn_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_352]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_352]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) ~[hadoop-client-api-3.2.1-amzn-8.jar:?]
    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:928) ~[spark-yarn_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0]
    at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) ~[spark-yarn_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0]
Caused by: org.apache.spark.SparkUserAppException: User application exited with 1
    at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:111) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0]
    at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_352]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_352]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_352]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_352]
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:742) ~[spark-yarn_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0]
22/12/20 00:52:24 INFO ApplicationMaster: Deleting staging directory hdfs://ip-10-0-7-133.us-west-2.compute.internal:8020/user/hadoop/.sparkStaging/application_1671497445713_0001
22/12/20 00:52:25 INFO ShutdownHookManager: Shutdown hook called

故障根源与修复 #

核心问题 #

你错误地将Python脚本（.py）配置为JAR文件类型的任务。Spark的JAR步骤是为Scala/Java编译的JAR包设计的，当系统尝试用JVM加载非JAR格式的Python脚本时，会触发执行失败，最终抛出SparkUserAppException: User application exited with 1。

修复步骤 #

1. 删除EMR集群中失败的错误步骤
2. 重新添加正确的步骤：
   • 步骤类型选择Spark应用程序（Python）
   • 在「Python脚本位置」输入S3上的.py文件完整路径
   • 按需填写执行参数、Spark配置等，保持与官方教程一致

新手排查技巧 #

• 查看详细日志：在EMR步骤详情页点击「查看日志」，重点看stderr日志，里面会有更直接的错误提示（比如“无法解析JAR文件”）
• 验证文件权限：确保EMR集群的IAM角色拥有访问目标S3路径的权限
• 确认环境支持：EMR默认Spark集群预装了PySpark，无需额外配置Python环境（除非自定义集群）

内容的提问来源于stack exchange，提问作者Kei