PyInstaller打包Python项目为单可执行文件后运行失败求助

问题背景 #

使用PyInstaller将包含Torch、TorchVision等依赖的Python项目打包为单可执行文件，原项目运行正常，但执行打包后的文件时出现一系列警告与错误。

项目依赖 #

Pillow  9.3.0   9.3.0
altgraph    0.17.3  0.17.3
certifi 2022.9.24   2022.9.24
charset-normalizer  2.1.1   3.0.1
contourpy   1.0.6   1.0.6
cycler  0.11.0  0.11.0
fonttools   4.38.0  4.38.0
idna    3.4 3.4
kiwisolver  1.4.4   1.4.4
matplotlib  3.6.2   3.6.2
numpy   1.23.5  1.23.5
nvidia-cublas-cu11  11.10.3.66  11.11.3.6
nvidia-cuda-nvrtc-cu11  11.7.99 11.8.89
nvidia-cuda-runtime-cu11    11.7.99 11.8.89
nvidia-cudnn-cu11   8.5.0.96    8.6.0.163
opencv-python   4.6.0.66    4.6.0.66
packaging   21.3    21.3
pip 21.3.1  22.3.1
pyinstaller 5.6.2   5.6.2
pyinstaller-hooks-contrib   2022.13 2022.13
pyparsing   3.0.9   3.0.9
python-dateutil 2.8.2   2.8.2
requests    2.28.1  2.28.1
setuptools  60.2.0  65.6.3
six 1.16.0  1.16.0
torch   1.13.0  1.13.0
torchvision 0.14.0  0.14.0
typing-extensions   4.4.0   4.4.0
urllib3 1.26.13 1.26.13
wheel   0.38.4  0.38.4

执行报错信息 #

[19507] WARNING: file already exists but should not: /tmp/_MEIF8RVEq/torch/_C.cpython-39-x86_64-linux-gnu.so
[19507] WARNING: file already exists but should not: /tmp/_MEIF8RVEq/torch/_C_flatbuffer.cpython-39-x86_64-linux-gnu.so
torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 
torch/_jit_internal.py:839: UserWarning: Unable to retrieve source for @torch.jit._overload function: <function _DenseLayer.forward at 0x7fc0c005b8b0>.  warnings.warn(
torch/_jit_internal.py:839: UserWarning: Unable to retrieve source for @torch.jit._overload function: <function _DenseLayer.forward at 0x7fc0c006fb80>.  warnings.warn(
torch/serialization.py:834: UserWarning: Couldn't retrieve source code for container of type RRDBNet. It won't be checked for correctness upon loading.  warnings.warn("Couldn't retrieve source code for container of "
torch/serialization.py:868: SourceChangeWarning: source code of class 'torch.nn.modules.conv.Conv2d' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.  warnings.warn(msg, SourceChangeWarning)
torch/serialization.py:868: SourceChangeWarning: source code of class 'torch.nn.modules.container.Sequential' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.  warnings.warn(msg, SourceChangeWarning)
torch/serialization.py:834: UserWarning: Couldn't retrieve source code for container of type RRDB. It won't be checked for correctness upon loading.  warnings.warn("Couldn't retrieve source code for container of "
torch/serialization.py:834: UserWarning: Couldn't retrieve source code for container of type ResidualDenseBlock. It won't be checked for correctness upon loading.  warnings.warn("Couldn't retrieve source code for container of "
torch/serialization.py:868: SourceChangeWarning: source code of class 'torch.nn.modules.activation.LeakyReLU' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.  warnings.warn(msg, SourceChangeWarning)

原因分析 #

1. Torch核心库重复打包：PyInstaller解析Torch依赖时，因钩子逻辑或多版本干扰，重复复制了torch/_C.so等核心二进制文件，导致临时目录文件冲突。
2. TorchVision扩展未被正确收集：TorchVision的image模块依赖的底层扩展库未被PyInstaller识别并打包，导致加载失败。
3. 模型加载时的源码路径问题：打包后PyInstaller修改了Python的源码检索路径，Torch JIT加载自定义模型（如RRDBNet）时无法找到对应源码；同时，CUDA相关依赖版本不一致，导致Torch模块源码误判为已修改。

解决方案 #

1. 解决Torch文件重复问题 #

• 清理缓存后重新打包，避免旧缓存文件干扰：

  pyinstaller --onefile --clean your_script.py
  

• 若使用虚拟环境，确保打包时激活虚拟环境，避免系统中多版本Torch的影响。
• 生成并修改spec文件，排除重复模块：
  先生成spec文件：

  pyi-makespec --onefile your_script.py
  

  在spec文件的excludes列表中添加重复相关的Torch模块，例如：

  excludes=['torch._C', 'torch._C_flatbuffer']
  

2. 修复TorchVision扩展加载失败 #

• 升级PyInstaller及钩子库至最新版本，新版本钩子已优化TorchVision的打包支持：

  pip install --upgrade pyinstaller pyinstaller-hooks-contrib
  

• 手动添加TorchVision扩展库到spec文件：
  先在Python中找到扩展库路径：

  import torchvision
  import os
  print(os.path.join(os.path.dirname(torchvision.__file__), 'io', 'image.so')) # Linux平台后缀为.so，Windows为.pyd
  

  然后在spec文件的binaries中添加该路径：

  binaries=[
      ("/path/to/torchvision/io/image.so", "torchvision/io")
  ]
  

  再用spec文件打包：pyinstaller your_script.spec

3. 处理模型加载的警告 #

• 禁用相关警告（若不影响功能）：
  在脚本开头添加：

  import torch
  import warnings
  from torch.serialization import SourceChangeWarning
  
  torch.nn.Module.dump_patches = False
  warnings.filterwarnings("ignore", category=UserWarning)
  warnings.filterwarnings("ignore", category=SourceChangeWarning)
  

• 确保自定义模型模块被正确打包：
  在spec文件的hiddenimports中添加模型所在模块：

  hiddenimports=['your_module.rrdbnet'] # 替换为实际的模型模块路径
  

• 加载模型时指定设备，避免路径干扰：

  model = torch.load('your_model.pth', map_location=torch.device('cpu')) # 或根据需求指定cuda设备
  

4. 统一CUDA依赖版本 #

项目中nvidia相关库存在版本差异（如nvidia-cublas-cu11有11.10和11.11版本），统一这些依赖版本，避免Torch加载时出现冲突：

pip install nvidia-cublas-cu11==11.11.3.6 nvidia-cuda-nvrtc-cu11==11.8.89 nvidia-cuda-runtime-cu11==11.8.89 nvidia-cudnn-cu11==8.6.0.163

内容的提问来源于stack exchange，提问作者justarandomuser1