在Jupyter Notebook中创建PySpark的SparkSession时出错

执行代码spark=SparkSession.builder.appName('Srikanth').getOrCreate()时出现如下错误，导致SparkSession无法初始化：

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_592\618625653.py in <module>
----> 1 spark=SparkSession.builder.appName('Srikanth').getOrCreate()

~\anaconda3\lib\site-packages\pyspark\sql\session.py in getOrCreate(self)
    267                         sparkConf.set(key, value)
    268                     # This SparkContext may be an existing one.
--> 269                     sc = SparkContext.getOrCreate(sparkConf)
    270                     # Do not update `SparkConf` for existing `SparkContext`, as it's shared
    271                     # by all sessions.

~\anaconda3\lib\site-packages\pyspark\context.py in getOrCreate(cls, conf)
    481         with SparkContext._lock:
    482             if SparkContext._active_spark_context is None:
--> 483                 SparkContext(conf=conf or SparkConf())
    484             assert SparkContext._active_spark_context is not None
    485             return SparkContext._active_spark_context

~\anaconda3\lib\site-packages\pyspark\context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls, udf_profiler_cls)
    193             )
    194 
--> 195         SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
    196         try:
    197             self._do_init(

~\anaconda3\lib\site-packages\pyspark\context.py in _ensure_initialized(cls, instance, gateway, conf)
    415         with SparkContext._lock:
    416             if not SparkContext._gateway:
--> 417                 SparkContext._gateway = gateway or launch_gateway(conf)
    418                 SparkContext._jvm = SparkContext._gateway.jvm
    419 

~\anaconda3\lib\site-packages\pyspark\java_gateway.py in launch_gateway(conf, popen_kwargs)
    104 
    105             if not os.path.isfile(conn_info_file):
--> 106                 raise RuntimeError("Java gateway process exited before sending its port number")
    107 
    108             with open(conn_info_file, "rb") as info:

RuntimeError: Java gateway process exited before sending its port number

解决方案： #

• 检查Java环境安装与配置

  1. 确认已安装Java 8或11（Spark对这两个版本兼容性最优）
  2. 打开命令行执行java -version和javac -version，验证Java是否正常运行
  3. 配置JAVA_HOME环境变量，指向JDK安装目录，并将%JAVA_HOME%\bin添加到系统PATH中（Windows）

• 验证PySpark与Java版本兼容性

  • 通过pip show pyspark查看当前PySpark版本，对照Spark官方文档确认支持的Java版本（例如Spark 3.x适配Java 8/11，Spark 2.x仅支持Java 8）

• 清理临时文件并重启环境

  1. 删除~\AppData\Local\Temp目录下的ipykernel临时文件
  2. 完全关闭Jupyter Notebook及相关进程，重新启动后再尝试创建SparkSession

• 显式指定Java路径（可选）
  如果环境变量配置仍不生效，可在代码中手动指定JDK路径：

  from pyspark.sql import SparkSession
  from pyspark.conf import SparkConf
  
  conf = SparkConf()
  conf.set("spark.driver.extraJavaOptions", "-Djava.home=C:/Program Files/Java/jdk1.8.0_381")  # 替换为实际JDK路径
  spark = SparkSession.builder.config(conf=conf).appName('Srikanth').getOrCreate()
  

• 排查Anaconda环境冲突

  1. 在Anaconda Prompt中直接执行pyspark命令，检查Spark Shell是否能正常启动
  2. 确认Anaconda环境中无冲突的Java依赖，必要时可创建新的虚拟环境单独安装PySpark

内容的提问来源于stack exchange，提问作者BOREDDY SRIKANTH REDDY