VS Code中Spark Session自动终止问题求助

问题场景 #

我在VS Code中运行以下Python脚本启动Spark Session：

from pyspark.sql import SparkSession
from pyspark.sql.types import *


spark = SparkSession.builder \
    .appName("test") \
    .getOrCreate()
    
spark.sparkContext.setLogLevel("DEBUG")

脚本未调用spark.stop()，但Spark Session启动后立即自动终止，日志显示Spark进程以exitCode 0停止，所有资源被清理：

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/11/04 01:26:56 DEBUG PythonGatewayServer: Exiting due to broken pipe from Python driver
24/11/04 01:26:56 INFO SparkContext: Invoking stop() from shutdown hook
24/11/04 01:26:56 INFO SparkContext: SparkContext is stopping with exitCode 0.
PS E:\graduation_project> 24/11/04 01:26:56 INFO SparkUI: Stopped Spark web UI at http://DESKTOP-CCJBLEA:4040
24/11/04 01:26:56 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
24/11/04 01:26:56 INFO MemoryStore: MemoryStore cleared
24/11/04 01:26:56 INFO BlockManager: BlockManager stopped
24/11/04 01:26:56 INFO BlockManagerMaster: BlockManagerMaster stopped
24/11/04 01:26:56 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
24/11/04 01:26:56 INFO SparkContext: Successfully stopped SparkContext
24/11/04 01:26:56 INFO ShutdownHookManager: Shutdown hook called
24/11/04 01:26:56 INFO ShutdownHookManager: Deleting directory C:\Users\Admin\AppData\Local\Temp\spark-cbe85e4c-a72a-4c3f-96b0-57564457230a\pyspark-37a42a15-2131-4bc3-a23c-e73f551df5ef
24/11/04 01:26:56 INFO ShutdownHookManager: Deleting directory C:\Users\Admin\AppData\Local\Temp\spark-cbe85e4c-a72a-4c3f-96b0-57564457230a
24/11/04 01:26:56 INFO ShutdownHookManager: Deleting directory C:\Users\Admin\AppData\Local\Temp\spark-d13033f1-7d8e-4e90-aadd-3a6fc6a3e502
24/11/04 01:26:56 DEBUG ShutdownHookManager: Completed shutdown in 0.146 seconds; Timeouts: 0
SUCCESS: The process with PID 9836 (child process of PID 15948) has been terminated.
SUCCESS: The process with PID 15948 (child process of PID 4180) has been terminated.
SUCCESS: The process with PID 4180 (child process of PID 3768) has been terminated.

但在CMD中直接运行pyspark命令一切正常：

C:\Users\Admin>pyspark
Python 3.11.8 (tags/v3.11.8:db85d51, Feb  6 2024, 22:03:32) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/11/04 07:28:11 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.5.3
      /_/

Using Python version 3.11.8 (tags/v3.11.8:db85d51, Feb  6 2024 22:03:32)
Spark context Web UI available at http://DESKTOP-CCJBLEA.mshome.net:4041
Spark context available as 'sc' (master = local[*], app id = local-1730680091248).
SparkSession available as 'spark'.
>>>

环境配置：Spark 3.5.3、JDK 17、Hadoop 3.3.5

尝试添加input阻塞后，Spark UI可维持运行，但终端未显示Spark欢迎图标：

from pyspark.sql import SparkSession
from pyspark.sql.types import *


spark = SparkSession.builder \
    .appName("test") \
    .getOrCreate()
    
input("Press Enter to exit...")
    
spark.sparkContext.setLogLevel("DEBUG")

运行结果：

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Press Enter to exit...

解决方案 #

核心原因 #

VS Code中直接运行脚本时，脚本执行完所有代码后会立即退出进程，Spark内置的Shutdown Hook会自动触发stop()清理资源；而pyspark命令启动的是交互式Shell，进程会保持活跃等待用户输入，因此Session不会终止。

针对性解决方法 #

• 方法一：使用VS Code交互式窗口
  按下Ctrl+Shift+P，选择"Python: 启动交互式窗口"，将代码逐行或整块粘贴执行。这种模式和CMD的pyspark效果完全一致，进程保持活跃，能正常显示Spark欢迎信息，Session也不会自动终止。

• 方法二：优化脚本阻塞逻辑与日志输出
  如果必须通过脚本文件运行，可调整日志级别并添加阻塞逻辑，确保欢迎信息正常显示：

  from pyspark.sql import SparkSession
  import sys
  
  # 初始化Spark Session
  spark = SparkSession.builder \
      .appName("test") \
      .getOrCreate()
  
  # 设置日志级别为INFO，确保欢迎信息输出
  spark.sparkContext.setLogLevel("INFO")
  # 强制刷新输出，避免信息被缓存
  sys.stdout.flush()
  
  # 阻塞进程，保持Session运行
  input("Press Enter to stop Spark Session...")
  
  # 手动终止Session（可选，也可依赖Shutdown Hook自动清理）
  spark.stop()
  

  运行后终端会显示Spark欢迎图标和信息，Session保持运行直到用户输入回车。

• 方法三：配置VS Code运行参数
  新建或修改.vscode/launch.json配置文件，设置脚本在集成终端中运行：

  {
      "version": "0.2.0",
      "configurations": [
          {
              "name": "Python: Current File",
              "type": "python",
              "request": "launch",
              "program": "${file}",
              "console": "integratedTerminal",
              "justMyCode": true
          }
      ]
  }
  

  配合脚本中的阻塞逻辑，终端输出效果会更接近CMD的pyspark。

内容的提问来源于stack exchange，提问作者DucAnh_Ng