如何在macOS Sonoma的Jupyter Notebook中运行含多进程的自定义模块？

问题 #

自行编写的模块中，有个用concurrent futures实现多进程的函数，出现如下运行差异：

• macOS系统：在独立.py文件中运行无报错，但在Jupyter Notebook中运行报错
• Ubuntu系统：无论是独立.py文件还是Jupyter Notebook中运行均无异常

调用代码如下：

import HSTI as hsti
import numpy as np

if __name__ == '__main__':
    lams = np.linspace(8,16,256)*1e-6
    d = np.linspace(3,13,128)*1e-6
    FPI = hsti.FPI(d,lams,297)

Jupyter Notebook中报错信息：

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 131, in _main
    prepare(preparation_data)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 246, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 290, in run_path
  File "<frozen runpy>", line 254, in _get_code_from_file
FileNotFoundError: [Errno 2] No such file or directory: '/Users/madslarsen/Dropbox/HSTI/spectral_calibration/HSTI/spectral_calibration/tmm_test.ipynb'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 131, in _main
    prepare(preparation_data)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 246, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 290, in run_path
  File "<frozen runpy>", line 254, in _get_code_from_file
FileNotFoundError: [Errno 2] No such file or directory: '/Users/madslarsen/Dropbox/HSTI/spectral_calibration/HSTI/spectral_calibration/tmm_test.ipynb'

解决建议 #

核心原因 #

macOS上Python多进程默认采用spawn启动方式，子进程会尝试重新导入主模块，但Jupyter Notebook的.ipynb文件无法被runpy.run_path直接解析；同时报错中出现重复路径（/HSTI/spectral_calibration重复），说明Notebook环境下的路径解析逻辑出现异常。

具体方案 #

1. 封装多进程逻辑到独立模块
   把HSTI模块中涉及多进程的代码拆分为单独的函数/类，放在独立的.py文件中，确保子进程启动时能正确导入该模块，避免直接依赖Notebook环境。

2. 修正Notebook调用逻辑与工作目录
   在Notebook中，将调用逻辑封装到单独函数内，用if __name__ == '__main__':包裹，同时手动设置正确的工作目录，避免路径重复问题：

   import HSTI as hsti
   import numpy as np
   import os
   
   # 手动设置正确的工作目录，解决路径重复问题
   os.chdir('/Users/madslarsen/Dropbox/HSTI/spectral_calibration')
   
   def run_fpi():
       lams = np.linspace(8,16,256)*1e-6
       d = np.linspace(3,13,128)*1e-6
       FPI = hsti.FPI(d,lams,297)
       return FPI
   
   if __name__ == '__main__':
       result = run_fpi()
   

3. 切换多进程启动方式为fork
   在Notebook开头添加代码，强制使用fork启动方式（与Ubuntu默认一致），避免子进程重新导入模块的问题：

   import multiprocessing
   multiprocessing.set_start_method('fork')
   

   注意：fork方式在Python 3.10+的macOS上可能与部分GUI库存在兼容性问题，但纯计算场景下可正常使用。

4. 检查模块内的路径与序列化问题
   确保HSTI模块中使用ProcessPoolExecutor时，传入的函数可被序列化，且无硬编码路径。若存在硬编码路径，改为动态获取：

   import os
   # 动态获取模块所在目录
   base_dir = os.path.dirname(os.path.abspath(__file__))
   # 用base_dir拼接其他路径，替代硬编码
   

内容的提问来源于stack exchange，提问作者madsnibe