配置Spark内容根路径后PySpark报TypeError的解决方法

在PyCharm中设置项目内容根路径为C:/spark/python后，运行PySpark代码时出现如下错误：

C:\Users\nande\PycharmProjects\Big_Data_Project_USA_Pharma\venv\Scripts\python.exe C:\Users\nande\PycharmProjects\Big_Data_Project_USA_Pharma\driver.py 
Traceback (most recent call last):
  File "C:\Users\nande\PycharmProjects\Big_Data_Project_USA_Pharma\driver.py", line 2, in <module>
    from create_spark import get_spark_object
  File "C:\Users\nande\PycharmProjects\Big_Data_Project_USA_Pharma\create_spark.py", line 1, in <module>
    from pyspark.sql import SparkSession
  File "C:\spark\python\pyspark\__init__.py", line 51, in <module>
    from pyspark.context import SparkContext
  File "C:\spark\python\pyspark\context.py", line 31, in <module>
    from pyspark import accumulators
  File "C:\spark\python\pyspark\accumulators.py", line 97, in <module>
    from pyspark.serializers import read_int, PickleSerializer
  File "C:\spark\python\pyspark\serializers.py", line 71, in <module>
    from pyspark import cloudpickle
  File "C:\spark\python\pyspark\cloudpickle.py", line 145, in <module>
    _cell_set_template_code = _make_cell_set_template_code()
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\spark\python\pyspark\cloudpickle.py", line 126, in _make_cell_set_template_code
    return types.CodeType(
           ^^^^^^^^^^^^^^^
TypeError: code expected at least 16 arguments, got 15

Process finished with exit code 1

当前环境信息 #

• Java版本：1.8.0_251
• Python版本：3.11.5
• PyCharm中PySpark版本：3.4.1

解决方案 #

这个错误的核心原因是Python 3.11与PySpark 3.4.1不兼容——Python 3.11对types.CodeType的参数结构做了调整，而PySpark 3.4.1的内置cloudpickle模块未适配该变化。同时，错误配置内容根路径直接指向Spark自带的pyspark模块，会绕过虚拟环境的兼容版本，加剧问题。

解决步骤：

1. 调整Python版本到兼容范围

   • 在PyCharm中进入File > Settings > Project: [项目名] > Python Interpreter
   • 点击右上角齿轮图标，选择Add，创建基于Python 3.8~3.10的虚拟环境（这几个版本与PySpark 3.4.1兼容性最佳）
   • 切换到新虚拟环境后，重新安装PySpark 3.4.1：pip install pyspark==3.4.1

2. 移除错误的内容根路径配置

   • 进入File > Settings > Project: [项目名] > Project Structure
   • 找到已添加的C:/spark/python内容根路径，点击减号移除
   • 保存设置，让PyCharm优先加载虚拟环境中安装的兼容pyspark模块

3. 验证修复效果
   运行以下测试代码确认问题解决：

from pyspark.sql import SparkSession

if __name__ == "__main__":
    spark = SparkSession.builder.appName("CompatibilityTest").getOrCreate()
    print("Spark会话创建成功")
    spark.stop()

内容的提问来源于stack exchange，提问作者Nandy