Apache Spark与PySpark运行异常：简单Python脚本触发报错求助

问题重现 #

已完成Apache Spark安装，PySpark解释器可正常启动，但执行如下测试脚本时，调用df.show()触发Python worker exited unexpectedly (crashed)异常：

from pyspark.sql import SparkSession

# Create a Spark session
spark = SparkSession.builder \
    .appName("SimpleApp") \
    .getOrCreate()

# Sample data
data = [("Alice", 25), ("Bob", 30), ("Charlie", 35)]

# Create a DataFrame from the data
df = spark.createDataFrame(data, ["Name", "Age"])

# Show the DataFrame content
df.show()

# Stop the Spark session
spark.stop()

异常信息 #

执行df.show()后输出的核心异常：

25/02/05 10:15:06 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)
        at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:612)
        at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:594)
        ...
25/02/05 10:15:06 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job
Traceback (most recent call last):
  File "<python-input-4>", line 1, in <module>
    df.show()
    ~~~~~~~^^
  File "D:\Tools2\spark-3.5.4-bin-hadoop3\python\pyspark\sql\dataframe.py", line 947, in show
    print(self._show_string(n, truncate, vertical))
          ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  ...
py4j.protocol.Py4JJavaError: An error occurred while calling o56.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (MDXN01072079.mshome.net executor driver): org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)

环境配置信息 #

环境检查脚本：

#!/usr/bin/bash
which python
python --version
echo $PYSPARK_PYTHON
echo $SPARK_HOME
which spark-shell
echo $HADOOP_HOME
which winutils

执行结果：

/c/Python313/python
Python 3.13.1
C:\Python313\python.exe
D:\Tools2\spark-3.5.4-bin-hadoop3
/c/Python313/Scripts/spark-shell
D:\Tools2\hadoop
/d/Tools2/hadoop/bin/winutils

解决方案 #

1. 降级Python版本到Spark兼容范围 #

Spark 3.5.4官方仅支持Python 3.8~3.12，Python 3.13存在兼容性问题，这是导致Worker崩溃的核心原因：

• 安装Python 3.12.x版本
• 更新PYSPARK_PYTHON环境变量指向新Python路径（如C:\Python312\python.exe）
• 调整系统PATH优先级，确保默认Python为3.12版本

2. 统一路径格式（Windows环境） #

当前环境混合了Windows路径和类Unix路径，可能导致Spark解析错误：

• 将PYSPARK_PYTHON修改为类Unix格式路径（如/c/Python312/python）
• 确保SPARK_HOME、HADOOP_HOME在系统环境和终端中路径格式一致，避免混合使用C:\和/c/格式

3. 验证权限与运行方式 #

Windows下权限限制可能导致Worker无法启动：

• 以管理员身份运行终端或PySpark
• 检查Spark和Python安装目录的权限，确保当前用户拥有读写、执行权限

4. 修复PySpark依赖 #

确保PySpark依赖库版本与Spark自带版本一致：

pip install pyspark==3.5.4 py4j==0.10.9.7

内容的提问来源于stack exchange，提问作者petalumaboy