PySpark集群模式下如何设置Python驱动路径解决版本兼容问题？

我的程序在client模式下运行正常，但切换到cluster模式时运行失败。排查YARN日志后确认是集群节点的Python版本不兼容导致的。

我尝试通过spark-submit命令配置spark.yarn.appMasterEnv.PYSPARK_PYTHON指定Python路径，提交命令如下：

spark-submit --master yarn --deploy-mode cluster --num-executors 10 --executor-cores 3 --driver-memory 50G --executor-memory 20G \
--conf spark.dynamicAllocation.enabled=false  \
--conf spark.kryoserializer.buffer.max=1024 --conf spark.yarn.keytab=keytab_path --conf spark.yarn.principal=${10} \
--conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=/bin/python3
--jars path_to_jars \
--py-files Pipeline.egg-info,<path>/app.py <application_path>/app.py arguments

但仍出现报错，错误日志如下：

22/08/04 06:09:34 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
22/08/04 06:09:34 INFO yarn.ApplicationMaster: Waiting for spark context initialization...
22/08/04 06:09:34 ERROR yarn.ApplicationMaster: User application exited with status 1
22/08/04 06:09:34 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: User application exited with status 1)
22/08/04 06:09:34 ERROR yarn.ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
        at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:447)
        at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:275)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:805)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:804)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
        at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:804)
        at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: org.apache.spark.SparkUserAppException: User application exited with 1
        at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:106)
        at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
22/08/04 06:09:34 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://test-scc/user/tst_rdip_cross/.sparkStaging/application_1643123069214_48871
22/08/04 06:09:35 INFO util.ShutdownHookManager: Shutdown hook called

解决方案 #

1. 同时配置Executor和AppMaster的Python路径 #

仅配置AppMaster的Python环境不够，Executor节点也需要指定相同的Python路径，否则会使用系统默认版本导致不兼容。需要添加两个关键配置：

• spark.yarn.appMasterEnv.PYSPARK_PYTHON：指定AppMaster使用的Python路径
• spark.executorEnv.PYSPARK_PYTHON：指定所有Executor节点使用的Python路径

修改后的提交命令：

spark-submit --master yarn --deploy-mode cluster --num-executors 10 --executor-cores 3 --driver-memory 50G --executor-memory 20G \
--conf spark.dynamicAllocation.enabled=false  \
--conf spark.kryoserializer.buffer.max=1024 --conf spark.yarn.keytab=keytab_path --conf spark.yarn.principal=${10} \
--conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=/bin/python3 \
--conf spark.executorEnv.PYSPARK_PYTHON=/bin/python3 \
--jars path_to_jars \
--py-files Pipeline.egg-info,<path>/app.py <application_path>/app.py arguments

2. 验证Python路径的有效性 #

确保/bin/python3在所有YARN节点（ResourceManager、NodeManager节点）都存在，且版本与本地开发环境一致。可在集群节点执行以下命令验证：

which python3
python3 --version

若路径不存在，替换为集群实际的Python3路径（如/usr/local/bin/python3）。

3. 检查第三方依赖的兼容性 #

如果程序依赖第三方Python包，需确保所有Executor节点已安装对应包，或通过--py-files将依赖打包上传：

1. 导出依赖列表：pip freeze > requirements.txt
2. 下载依赖包：pip download -d ./deps -r requirements.txt
3. 打包依赖：zip -r deps.zip ./deps
4. 提交时添加：--py-files deps.zip

4. 获取更详细的错误日志 #

当前日志未显示具体Python错误，可通过以下方式排查：

• 通过YARN UI找到对应应用，点击「Logs」查看完整的stderr日志，里面包含Python脚本执行的具体报错信息
• 添加配置spark.yarn.maxAppAttempts=1，避免应用重试覆盖第一次失败的日志

内容的提问来源于stack exchange，提问作者Akhil