如何用Python多进程池调用Julia函数？遇断言失败问题求助

问题场景 #

单线程通过juliacall调用Julia函数/脚本正常，但使用Python multiprocessing Pool并行调用时出现断言错误：

python: /buildworker/worker/package_linux64/build/src/debuginfo.cpp:1634: void register_eh_frames(uint8_t*, size_t): Assertion `end_ip != 0' failed.

改用ThreadPoolExecutor线程池则触发Segmentation fault，错误与Julia官方issue #44969高度相似。

复现代码 #

Python代码

import os
os.environ['PYTHON_JULIAPKG_EXE'] = "/home/user/.juliaup/bin/julia"
os.environ['PYTHON_JULIAPKG_OFFLINE'] = 'yes'
os.environ['PYTHON_JULIAPKG_PROJECT'] = '/home/user/julia/environments/v1.6/'

from juliacall import Main as jl, convert as jlconvert

from multiprocessing import Pool
from tqdm import tqdm

import ipdb

def init_worker():
    import os
    os.environ['PYTHON_JULIAPKG_EXE'] = "/home/user/juliaup/bin/julia"
    os.environ['PYTHON_JULIAPKG_OFFLINE'] = 'yes'
    os.environ['PYTHON_JULIAPKG_PROJECT'] = '/home/user/.julia/environments/v1.6/'

    from juliacall import Main as jl, convert as jlconvert

    print('in init_worker()...')
    jl.seval('using Pkg')
    jl.seval('Pkg.status()')
    print('...done')

def compute(jobid):
    print(f'in main({jobid})...')

    jl.seval('include("test_julia_simple.jl")')
    print('...done')
    return

def main():
    njobs = 10
    #start pool with init_worker() as initializer
    
    with Pool(2, initializer=init_worker) as p, tqdm(total=njobs) as pbar:
        res = []
        for jid in range(njobs):
            res.append(p.apply_async(compute, (jid,)))
        for r in res:
            r.get()
            pbar.update(1)


if __name__ == "__main__":
    main()

Julia脚本test_julia_simple.jl

for i in 1:10
    println(i)
end
1+2

环境信息 #

$ python --version
Python 3.9.7
$ pip freeze | grep julia
juliacall==0.9.10
juliapkg==0.1.9

$ julia --version
The latest version of Julia in the `1.6` channel is 1.6.7+0.x64.linux.gnu. You currently have `1.6.6+0~x64` installed. Run:
  juliaup update
to install Julia 1.6.7+0.x64.linux.gnu and update the `1.6` channel to that version.
julia version 1.6.6

调试与解决方法 #

1. 升级Julia版本 #

当前使用的Julia 1.6.6存在已知的多进程兼容问题，执行以下命令升级到同频道的最新版本（1.6.7）：

juliaup update

升级后重新测试并行调用，多数情况下该断言错误会被修复。

2. 修改multiprocessing启动方式 #

Unix系统下multiprocessing默认使用fork模式，而Julia运行时不支持fork后的进程复用，需改用spawn模式启动进程池：

from multiprocessing import get_context

def main():
    njobs = 10
    # 使用spawn上下文初始化进程池
    with Pool(2, initializer=init_worker, context=get_context('spawn')) as p, tqdm(total=njobs) as pbar:
        res = []
        for jid in range(njobs):
            res.append(p.apply_async(compute, (jid,)))
        for r in res:
            r.get()
            pbar.update(1)

spawn模式会启动全新的Python进程，每个进程独立初始化Julia环境，避免fork导致的运行时冲突。

3. 优化Worker初始化逻辑 #

主进程不要提前导入juliacall，所有Julia相关的导入和初始化都放在init_worker中，防止主进程初始化Julia后被fork引发问题：
修改主进程开头代码：

import os
os.environ['PYTHON_JULIAPKG_EXE'] = "/home/user/.juliaup/bin/julia"
os.environ['PYTHON_JULIAPKG_OFFLINE'] = 'yes'
os.environ['PYTHON_JULIAPKG_PROJECT'] = '/home/user/julia/environments/v1.6/'

from multiprocessing import Pool, get_context
from tqdm import tqdm

4. 避免重复加载Julia脚本 #

在init_worker中提前加载脚本，而非每次调用compute时重复include，减少资源开销和潜在冲突：

def init_worker():
    import os
    os.environ['PYTHON_JULIAPKG_EXE'] = "/home/user/juliaup/bin/julia"
    os.environ['PYTHON_JULIAPKG_OFFLINE'] = 'yes'
    os.environ['PYTHON_JULIAPKG_PROJECT'] = '/home/user/.julia/environments/v1.6/'

    from juliacall import Main as jl
    print('in init_worker()...')
    jl.seval('using Pkg')
    jl.seval('Pkg.status()')
    jl.seval('include("test_julia_simple.jl")')  # 提前加载脚本
    print('...done')

5. 段错误深度调试 #

如果仍出现Segmentation fault，可使用gdb定位错误位置：

gdb --args python your_script.py
(gdb) run
# 发生段错误后执行
(gdb) bt

同时设置环境变量开启Julia调试日志：

export JULIA_DEBUG=all
python your_script.py

内容的提问来源于stack exchange，提问作者v.tralala