PySpark创建SparkSession时出现JVM方法不存在错误求助

问题场景 #

在VS Code中运行PySpark代码创建SparkSession时，触发py4j.protocol.Py4JError错误，提示org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTimeout不存在于JVM中。已确认PySpark与Spark版本一致，且代码和Jupyter Notebook中的教程代码完全一致。

报错核心信息 #

py4j.protocol.Py4JError: org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTimeout does not exist in the JVM

运行代码 #

import pyspark
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Practice").getOrCreate()

完整回溯信息 #

22/09/01 08:50:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Traceback (most recent call last):
  File "c:\Users\BERNARD JOSHUA\OneDrive\Desktop\Swinburne Computer Science\PySpark\pySpark_test.py", line 4, in <module>
    spark = SparkSession.builder.appName("Practice").getOrCreate()
  File "C:\Users\BERNARD JOSHUA\AppData\Local\Programs\Python\Python310\lib\site-packages\pyspark\sql\session.py", line 269, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "C:\Users\BERNARD JOSHUA\AppData\Local\Programs\Python\Python310\lib\site-packages\pyspark\context.py", line 483, in getOrCreate    
    SparkContext(conf=conf or SparkConf())
  File "C:\Users\BERNARD JOSHUA\AppData\Local\Programs\Python\Python310\lib\site-packages\pyspark\context.py", line 197, in __init__       
    self._do_init(
  File "C:\Users\BERNARD JOSHUA\AppData\Local\Programs\Python\Python310\lib\site-packages\pyspark\context.py", line 302, in _do_init       
    self._jvm.PythonUtils.getPythonAuthSocketTimeout(self._jsc)
  File "C:\Users\BERNARD JOSHUA\AppData\Local\Programs\Python\Python310\lib\site-packages\py4j\java_gateway.py", line 1547, in __getattr__ 
    raise Py4JError(
py4j.protocol.Py4JError: org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTimeout does not exist in the JVM
PS C:\Users\BERNARD JOSHUA\OneDrive\Desktop\Swinburne Computer Science\PySpark> SUCCESS: The process with PID 18428 (child process of PID 11272) has been terminated.
SUCCESS: The process with PID 11272 (child process of PID 16416) has been terminated.
SUCCESS: The process with PID 16416 (child process of PID 788) has been terminated.

解决方案 #

这个错误本质是Python端PySpark调用了JVM端Spark不存在的方法，常见解决办法如下：

• 检查环境变量配置
  确保SPARK_HOME指向正确的Spark安装目录（不是PySpark的Python包路径）；确认PYSPARK_PYTHON和PYSPARK_DRIVER_PYTHON指向当前使用的Python解释器路径；配置完成后重启VS Code让环境变量生效。

• 清理缓存并重新安装PySpark
  删除PySpark包目录下的临时缓存文件；卸载当前PySpark，重新安装与Spark版本完全匹配的PySpark：

  pip uninstall pyspark
  pip install pyspark==<你的Spark版本号>
  

• 手动指定Spark配置初始化
  通过SparkConf强制指定本地运行环境及相关配置：

  import pyspark
  from pyspark.sql import SparkSession
  from pyspark.conf import SparkConf
  
  conf = SparkConf() \
      .setAppName("Practice") \
      .setMaster("local[*]") \
      .set("spark.driver.host", "127.0.0.1")
  
  spark = SparkSession.builder.config(conf=conf).getOrCreate()
  

• 检查Java版本兼容性
  Spark对Java版本有明确要求（如Spark 3.x需Java 8或11），确保安装的Java版本符合当前Spark版本要求，且JAVA_HOME环境变量配置正确。

内容的提问来源于stack exchange，提问作者Bernard Joshua