如何在主Jupyter Notebook中并行调用多个子Notebook？

方法1：使用concurrent.futures.ProcessPoolExecutor #

利用Python进程池并行执行子Notebook，适合CPU密集型任务。借助IPython API在子进程中执行%run魔法命令：

from concurrent.futures import ProcessPoolExecutor
from IPython import get_ipython

def run_notebook(notebook_path):
    ip = get_ipython()
    ip.run_line_magic('run', notebook_path)

# 子Notebook列表
notebooks = [
    './Notebook1.ipynb',
    './Notebook2.ipynb',
    './Notebook3.ipynb',
    './Notebook4.ipynb',
    './Notebook5.ipynb'
]

# 并行执行，max_workers可根据CPU核心数调整
with ProcessPoolExecutor(max_workers=3) as executor:
    executor.map(run_notebook, notebooks)

注意事项：

• 子进程环境独立，主Notebook的全局变量不会被继承，子Notebook的输出也不会直接返回主Notebook，需通过文件、数据库等外部存储传递结果。
• 避免子Notebook读写同一文件，防止数据冲突。

方法2：使用multiprocessing模块 #

直接创建子进程运行每个Notebook，逻辑更直观：

import multiprocessing
from IPython import get_ipython

def run_notebook(notebook_path):
    ip = get_ipython()
    ip.run_line_magic('run', notebook_path)

if __name__ == '__main__':
    notebooks = [
        './Notebook1.ipynb',
        './Notebook2.ipynb',
        './Notebook3.ipynb',
        './Notebook4.ipynb',
        './Notebook5.ipynb'
    ]
    processes = []
    for nb in notebooks:
        p = multiprocessing.Process(target=run_notebook, args=(nb,))
        processes.append(p)
        p.start()
    
    # 等待所有进程执行完成
    for p in processes:
        p.join()

方法3：使用nbconvert命令行工具 #

通过jupyter nbconvert命令直接执行Notebook，结合subprocess实现并行，环境隔离性更好：

from concurrent.futures import ProcessPoolExecutor
import subprocess

def run_notebook_with_nbconvert(notebook_path):
    # 执行Notebook并覆盖原文件，可替换--inplace为--output指定新输出文件
    cmd = [
        'jupyter', 'nbconvert',
        '--execute',
        '--to', 'notebook',
        '--inplace',
        notebook_path
    ]
    subprocess.run(cmd, check=True)

notebooks = [
    './Notebook1.ipynb',
    './Notebook2.ipynb',
    './Notebook3.ipynb',
    './Notebook4.ipynb',
    './Notebook5.ipynb'
]

with ProcessPoolExecutor(max_workers=3) as executor:
    executor.map(run_notebook_with_nbconvert, notebooks)

内容的提问来源于stack exchange，提问作者Javi Torre