如何从PyInstaller生成的可执行文件中自动检测PySpark主目录路径？

问题背景 #

在本地开发环境中，我可以无需任何配置轻松运行PySpark应用，但在服务器上我们用PyInstaller打包成可执行文件部署时，发现PyInstaller没有把PySpark库的_internal文件夹包含进去，所以必须手动设置路径。

我的PyInstaller脚本片段 #

# -*- mode: python ; coding: utf-8 -*-

# Analysis for manage.py
a_manage = Analysis(
    ['manage.py'],
    pathex=['/app/app_name/app_name-backend-dev'],
    # 我尝试把.venv/lib/python3.11/site-packages添加到pathex，但没用
    binaries=[
        ('/usr/lib/x86_64-linux-gnu/libpython3.11.so.1.0', './_internal/libpython3.11.so.1.0')
    ],
    datas=[],
    hiddenimports=[
        # 我尝试添加PySpark的导入，但没用
        'pyspark', 'pyspark.sql', 'pyspark.sql.session', 'pyspark.sql.functions', 'pyspark.sql.types', 'pyspark.sql.column',
        
        
        'app_name2.apps', 'Crypto.Cipher', 'Crypto.Util.Padding', 'snakecase', 'cryptography.fernet',
        'cryptography.hazmat.primitives', 'cryptography.hazmat.primitives.kdf.pbkdf2', 'apscheduler.triggers.cron',
        'apscheduler.schedulers.background', 'apscheduler.events', 'oauth2_provider.contrib.rest_framework',
        'app_name.apps', 'app_name.role_permissions', 'django_filters.rest_framework', 'app_name.urls',
        'app_name.others.constants', 'app_name.models', 'app_name', 'sslserver'
    ],
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    noarchive=False,
)

pyz_manage = PYZ(a_manage.pure)

exe_manage = EXE(
    pyz_manage,
    a_manage.scripts,
    [],
    exclude_binaries=True,
    name='manage',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
)

coll_manage = COLLECT(
    exe_manage,
    a_manage.binaries,
    a_manage.datas,
    strip=False,
    upx=True,
    upx_exclude=[],
    name='manage',
)

运行可执行文件时的错误 #

Traceback (most recent call last):
  File "portal/operations/load_data/load_data.py", line 57, in start
  File "portal/pyspark/operations.py", line 498, in get_session
  File "pyspark/sql/session.py", line 497, in getOrCreate
  File "pyspark/context.py", line 515, in getOrCreate
  File "pyspark/context.py", line 201, in __init__
  File "pyspark/context.py", line 436, in _ensure_initialized
  File "pyspark/java_gateway.py", line 97, in launch_gateway
  File "subprocess.py", line 1026, in __init__
  File "subprocess.py", line 1955, in _execute_child
FileNotFoundError: [Errno 2] No such file or directory: '/home/rhythmflow/Desktop/Reconciliation/reconciliation-backend-v3/dist/manage/_internal/./bin/spark-submit'

我尝试的临时解决方法 #

我在Linux主目录创建了全局.venv，用pip install pyspark安装了PySpark，然后手动设置SPARK_HOME环境变量：

SPARK_HOME = /home/user_name/.venv/lib/python3.11/site-packages/pyspark

并在代码中这样使用：

SPARK_HOME = env_var("SPARK_HOME")
SparkSession.builder.appName(app_name).config("spark.home", SPARK_HOME).getOrCreate()

这个方法在开发环境能用，但我想简化流程，避免手动指定路径。

后续尝试的自动检测方法（失败） #

我尝试用代码自动获取PySpark路径：

import pyspark
SPARK_HOME = os.path.dirname(pyspark.__file__)

但遇到了以下错误，看来PySpark没有被正确打包进可执行文件：

Could not find valid SPARK_HOME while searching ['/home/user_name/Desktop/project_name/app_name/dist', '/home/user_name/Desktop/project_name/app_name/dist/manage/_internal/pyspark/spark-distribution', '/home/user_name/Desktop/project_name/app_name/dist/manage/_internal/pyspark', '/home/user_name/Desktop/project_name/app_name/dist/manage/_internal/pyspark/spark-distribution', '/home/user_name/Desktop/project_name/app_name/dist/manage/_internal/pyspark', '/home/user_name/Desktop/project_name/app_name/dist/manage']
Traceback (most recent call last):
  File "pyspark/find_spark_home.py", line 73, in _find_spark_home
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "wsgiref/handlers.py", line 137, in run
  File "django/contrib/staticfiles/handlers.py", line 80, in __call__
    return self.application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "django/core/handlers/wsgi.py", line 124, in __call__
    response = self.get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "django/core/handlers/base.py", line 140, in get_response
    response = self._middleware_chain(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "corsheaders/middleware.py", line 56, in __call__
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "portal/middleware.py", line 29, in __call__
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "portal/encdec_middleware.py", line 53, in __call__
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "portal/middleware.py", line 115, in __call__
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "django/views/decorators/csrf.py", line 65

解决方案 #

我来帮你分析一下问题的核心：PyInstaller默认不会完整打包PySpark的所有依赖文件（比如bin/spark-submit这类关键脚本、jars目录下的依赖包），所以自动检测路径的前提是先让PyInstaller正确把PySpark的整个目录结构包含进可执行文件中。咱们一步步来解决：

1. 修改PyInstaller配置，完整打包PySpark目录 #

在PyInstaller的Analysis配置中，通过datas参数把PySpark的完整安装目录复制到打包后的_internal文件夹下，这样所有必要的文件都会被包含：

# 替换你的Analysis里的datas配置
datas=[
    # 注意替换成你实际的PySpark安装路径，这里用venv的路径举例
    ("/home/user_name/.venv/lib/python3.11/site-packages/pyspark", "./_internal/pyspark")
],

这个配置会把PySpark的所有文件（包括bin、jars、python这些子目录）都复制到打包后的dist/manage/_internal/pyspark路径下。

2. 在代码中自动检测打包后的PySpark路径 #

现在PySpark已经被正确打包，我们可以通过PyInstaller提供的sys._MEIPASS变量（打包后的临时目录路径）来自动定位SPARK_HOME：

import os
import sys
import pyspark

def get_auto_spark_home():
    # 判断是否是PyInstaller打包后的运行环境
    if getattr(sys, 'frozen', False) and hasattr(sys, '_MEIPASS'):
        # 打包后的环境：PySpark在_internal/pyspark目录下
        return os.path.join(sys._MEIPASS, '_internal', 'pyspark')
    else:
        # 开发环境：直接用PySpark的安装目录
        return os.path.dirname(pyspark.__file__)

# 初始化SparkSession时使用自动获取的路径
spark = SparkSession.builder \
    .appName("your_app_name") \
    .config("spark.home", get_auto_spark_home()) \
    .getOrCreate()

sys._MEIPASS是PyInstaller打包后自动设置的环境变量，指向可执行文件解压后的临时工作目录，通过它我们能精准找到打包进去的PySpark目录。

3. 补充必要的Hidden Imports #

虽然我们已经打包了PySpark的目录，但还是要确保Python模块被PyInstaller正确识别，在hiddenimports中补充完整的PySpark依赖：

hiddenimports=[
    # 核心PySpark模块
    'pyspark', 'pyspark.sql', 'pyspark.sql.session', 'pyspark.sql.functions',
    'pyspark.sql.types', 'pyspark.sql.column', 'pyspark.context', 'pyspark.java_gateway',
    'pyspark.conf', 'pyspark.rdd', 'pyspark.serializers',
    # 你的其他项目依赖...
    'app_name2.apps', 'Crypto.Cipher', 'Crypto.Util.Padding', ...
],

为什么之前的尝试失败？ #

• 只添加pathex没用：pathex只是PyInstaller查找Python模块的路径，不会复制非Python文件（比如spark-submit脚本）。
• 只加hiddenimports不够：hiddenimports告诉PyInstaller要包含哪些Python模块，但PySpark运行需要的是整个目录结构的支持，不是光Python模块就行。
• 直接用os.path.dirname(pyspark.__file__)失败：因为PyInstaller没打包完整的PySpark目录，所以即使模块能导入，关键的bin目录也不存在，导致检测SPARK_HOME失败。

验证打包效果 #

打包完成后，你可以进入dist/manage/_internal目录，检查是否存在完整的pyspark文件夹，里面包含bin/spark-submit、jars等子目录，这样运行时就能自动找到正确的SPARK_HOME了。

备注：内容来源于stack exchange，提问作者Purushottam Nawale