创建PySpark DataFrame遇Py4JError/TypeError，求排查版本或环境问题

问题描述 #

创建基础PySpark DataFrame时触发Py4JError及TypeError（'JavaPackage' object is not callable），无法确定问题源于软件安装或版本不兼容，即使运行简单代码也失败。

环境配置 #

• Java 8
• Spark 3.3.2
• Python 3.9.13

报错代码 #

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('sparkdf').getOrCreate()
data = [["java", "dbms", "python"], 
        ["OOPS", "SQL", "Machine Learning"]]
columns = ["Subject 1", "Subject 2", "Subject 3"]
dataframe = spark.createDataFrame(data, columns)
dataframe.show()

错误栈信息 #

TypeError                                 Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_22308\2222183590.py in <module>
      5   StructField('lastname', StringType(), True)
      6   ])
----> 7 df = spark.createDataFrame(emptyRDD,schema)
      8 #df.printSchema()

~\AppData\Roaming\Python\Python39\site-packages\pyspark\sql\session.py in createDataFrame(self, data, schema, samplingRatio, verifySchema)
   1274                 data, schema, samplingRatio, verifySchema
   1275             )
-> 1276         return self._create_dataframe(
   1277             data, schema, samplingRatio, verifySchema  # type: ignore[arg-type]
   1278         )

~\AppData\Roaming\Python\Python39\site-packages\pyspark\sql\session.py in _create_dataframe(self, data, schema, samplingRatio, verifySchema)
   1318             rdd, struct = self._createFromLocal(map(prepare, data), schema)
   1319         assert self._jvm is not None
-> 1320         jrdd = self._jvm.SerDeUtil.toJavaArray(rdd._to_java_object_rdd())
   1321         jdf = self._jsparkSession.applySchemaToPythonRDD(jrdd.rdd(), struct.json())
   1322         df = DataFrame(jdf, self)


TypeError: 'JavaPackage' object is not callable

排查方案 #

• 检查SPARK_HOME环境变量是否正确配置，需指向Spark 3.3.2的安装目录，路径避免空格或特殊字符
• 确认PySpark通过Spark自带的Python包安装，而非单独执行pip install pyspark：Windows环境运行pip install %SPARK_HOME%\python，Linux/Mac运行pip install $SPARK_HOME/python，保证PySpark版本与Spark完全一致
• 验证JAVA_HOME环境变量，需准确指向Java 8的安装路径，且系统PATH中包含%JAVA_HOME%\bin（Windows）或$JAVA_HOME/bin（Linux/Mac）
• 检查PySpark版本：执行pip show pyspark，确认版本为3.3.2，避免多版本冲突
• 直接在Spark安装目录的bin文件夹下运行pyspark命令，启动交互式环境测试代码，排除Jupyter等第三方环境的配置干扰
• 查看Spark的jars目录，确认无关键依赖jar文件缺失，若缺失可重新解压完整的Spark安装包

内容的提问来源于stack exchange，提问作者Saravana Ayyappa