本地模式spark-submit指定虚拟环境无PySpark仍运行的原因咨询

我在本地模式下使用spark-submit调用Python虚拟环境运行程序，发现虚拟环境未安装PySpark但程序仍能正常运行，特咨询原因，以下是我的测试过程：

测试过程 #

• 验证spark-submit使用指定虚拟环境：卸载虚拟环境中的pytest后运行程序，触发ModuleNotFoundError，确认spark-submit确实调用了指定的虚拟环境Python解释器：

  PS C:\Users\demouser\Desktop\pytest_demo> spark-submit --master local[*] --conf spark.pyspark.python="C:\Users\demouser\Desktop\pytest_demo\pyspark-env\Scripts\python.exe" .\src\main.py 2>error.txt
  Traceback (most recent call last):
    File "C:/Users/demouser/Desktop/pytest_demo/./src/main.py", line 1, in <module>
      import pytest
  ModuleNotFoundError: No module named 'pytest'
  

  安装pytest的操作：

  PS C:\Users\demouser\Desktop\pytest_demo> .\pyspark-env\Scripts\Activate.ps1
  
  (pyspark-env) PS C:\Users\demouser\Desktop\pytest_demo> pip install pytest
  Collecting pytest
    Using cached pytest-7.1.2-py3-none-any.whl (297 kB)
  Requirement already satisfied: tomli>=1.0.0 in c:\users\demouser\desktop\pytest_demo\pyspark-env\lib\site-packages (from pytest) (2.0.1)
  Requirement already satisfied: pluggy<2.0,>=0.12 in c:\users\demouser\desktop\pytest_demo\pyspark-env\lib\site-packages (from pytest) (1.0.0)
  Requirement already satisfied: colorama in c:\users\demouser\desktop\pytest_demo\pyspark-env\lib\site-packages (from pytest) (0.4.5)
  Requirement already satisfied: py>=1.8.2 in c:\users\demouser\desktop\pytest_demo\pyspark-env\lib\site-packages (from pytest) (1.11.0)
  Requirement already satisfied: atomicwrites>=1.0 in c:\users\demouser\desktop\pytest_demo\pyspark-env\lib\site-packages (from pytest) (1.4.1)
  Requirement already satisfied: attrs>=19.2.0 in c:\users\demouser\desktop\pytest_demo\pyspark-env\lib\site-packages (from pytest) (22.1.0)
  Requirement already satisfied: packaging in c:\users\demouser\desktop\pytest_demo\pyspark-env\lib\site-packages (from pytest) (21.3)
  Requirement already satisfied: iniconfig in c:\users\demouser\desktop\pytest_demo\pyspark-env\lib\site-packages (from pytest) (1.1.1)
  Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in c:\users\demouser\desktop\pytest_demo\pyspark-env\lib\site-packages (from packaging->pytest) (3.0.9)
  Installing collected packages: pytest
  Successfully installed pytest-7.1.2
  (pyspark-env) PS C:\Users\demouser\Desktop\pytest_demo> deactivate
  

• 虚拟环境无PySpark但程序运行成功：安装pytest后，虚拟环境仍未通过pip安装PySpark，但程序正常运行并完成测试：

  PS C:\Users\demouser\Desktop\pytest_demo> spark-submit --master local[*] --conf spark.pyspark.python="C:\Users\demouser\Desktop\pytest_demo\pyspark-env\Scripts\python.exe" .\src\main.py 2>error.txt
  ============================= test session starts =============================
  platform win32 -- Python 3.8.12, pytest-7.1.2, pluggy-1.0.0
  rootdir: C:\Users\demouser\Desktop\pytest_demo\test, configfile: pytest.ini
  collected 6 items
  
  test\unit\test_factory.py::TestSparkSession::test_sparksession PASSED    [ 16%]
  test\unit\test_factory.py::TestFactory::test_five_dfs_count[Test_df_count0] PASSED [ 33%]
  test\unit\test_factory.py::TestFactory::test_five_dfs_count[Test_df_count1] PASSED [ 50%]
  test\unit\test_factory.py::TestFactory::test_five_dfs_count[Test_df_count2] PASSED [ 66%]
  test\unit\test_factory.py::TestFactory::test_five_dfs_count[Test_df_count3] PASSED [ 83%]
  test\unit\test_factory.py::TestFactory::test_five_dfs_count[Test_df_count4] PASSED [100%]
  
  ============================== warnings summary ===============================
  ....\Spark\spark-3.0.1-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\context.py:75
  unit/test_factory.py::TestSparkSession::test_sparksession
    C:\Spark\spark-3.0.1-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\context.py:75: DeprecationWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  
  -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
  ======================= 6 passed, 2 warnings in 26.90s ========================
  INFO     __main__:main.py: pytest session finished
  

• 系统已安装PySpark 3.0.1：本地系统中存在全局安装的PySpark版本：

  PS C:\Users\demouser\Desktop\pytest_demo> pyspark --version
  Welcome to
        ____              __
       / __/__  ___ _____/ /__
      _\ \/ _ \/ _ `/ __/  '_/
     /___/ .__/\_,_/_/ /_/\_\   version 3.0.1
        /_/
  
  Using Scala version 2.12.10, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_181
  Branch HEAD
  Compiled by user ubuntu on 2020-08-28T07:36:48Z
  Revision 2b147c4cd50da32fe2b4167f97c8142102a0510d
  Url https://gitbox.apache.org/repos/asf/spark.git
  Type --help for more information.
  

• 虚拟环境未安装PySpark：查看虚拟环境的已安装包列表，确认没有PySpark：

  PS C:\Users\demouser\Desktop\pytest_demo> .\pyspark-env\Scripts\Activate.ps1
  (pyspark-env) PS C:\Users\demouser\Desktop\pytest_demo> pip freeze
  atomicwrites==1.4.1
  attrs==22.1.0
  colorama==0.4.5
  iniconfig==1.1.1
  packaging==21.3
  pluggy==1.0.0
  py==1.11.0
  pyparsing==3.0.9
  pytest==7.1.2
  tomli==2.0.1
  (pyspark-env) PS C:\Users\demouser\Desktop\pytest_demo> python --version
  Python 3.8.12
  

原因分析 #

• spark-submit自动注入PySpark路径：使用spark-submit提交任务时，Spark的启动脚本会自动将全局Spark安装目录下的PySpark库（即测试警告中显示的C:\Spark\spark-3.0.1-bin-hadoop2.7\python\lib\pyspark.zip）添加到Python的sys.path中，无需虚拟环境通过pip安装PySpark，解释器就能找到PySpark模块。
• 虚拟环境负责非PySpark依赖：你通过spark.pyspark.python指定的虚拟环境Python解释器，主要用于运行程序并加载虚拟环境内的自定义依赖（比如pytest），PySpark本身由Spark的分发机制提供，不受虚拟环境包列表限制。
• 全局Spark是PySpark的来源：系统中安装的全局Spark 3.0.1就是spark-submit加载PySpark的源头，测试输出的警告路径也直接验证了这一点。

内容的提问来源于stack exchange，提问作者Vikat Thakur