使用Pool.starmap调用函数时遇tqdm与Jupyter冲突报错

问题根源 #

Jupyter交互式环境的输出机制与multiprocessing的子进程模型存在兼容性冲突：当用multiprocessing.Pool.starmap调用含tqdm的run_SpectraloASIS函数时，子进程中的tqdm会尝试更新主进程的进度条对象，触发Jupyter输出容器的重复close()操作，最终引发AttributeError: 'function' object has no attribute 'children'错误。

错误回溯示例 #

AttributeError: 'function' object has no attribute 'children'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/tqdm/std.py", line 1181, in close
self.display(msg=self.repr())
File "/usr/local/lib/python3.8/site-packages/tqdm/std.py", line 1081, in display
self.sp(self.str() if msg is None else msg)
File "/usr/local/lib/python3.8/site-packages/tqdm/notebook.py", line 287, in print_status
self.disp(bar_style='success')
File "/usr/local/lib/python3.8/site-packages/tqdm/notebook.py", line 260, in disp
self.out.container.children[-1].replace(new_bar)
AttributeError: 'function' object has no attribute 'children'

解决方案 #

1. 限制tqdm仅在主进程运行

修改run_SpectraloASIS函数，通过进程名称判断是否启用tqdm，避免子进程操作主进程的进度条对象：

import multiprocessing
from tqdm import tqdm

def run_SpectraloASIS(param1, param2):
    # 仅主进程初始化进度条
    is_main = multiprocessing.current_process().name == 'MainProcess'
    pbar = tqdm(total=100) if is_main else None
    
    # 你的Spectral oASIS特征选择逻辑
    # ...
    
    if is_main:
        pbar.update(1)
        pbar.close()
    return result

2. 使用tqdm原生多进程适配方法

替换multiprocessing.Pool.starmap为tqdm.contrib.concurrent.process_map，该方法已内置多进程与Jupyter环境的兼容处理：

from tqdm.contrib.concurrent import process_map

# 整理starmap所需的参数组为元组列表
params = [(p1_1, p2_1), (p1_2, p2_2), ...]
# 拆分参数组为独立的参数列表，适配process_map的传参格式
args1, args2 = zip(*params)

# 直接调用process_map，自动处理进度条与多进程
results = process_map(run_SpectraloASIS, args1, args2)

3. 改用spawn启动子进程

修改multiprocessing的启动方式为spawn，避免fork机制继承Jupyter的输出对象：

import multiprocessing

# 需在代码最开头设置启动方式
multiprocessing.set_start_method('spawn')

def run_SpectraloASIS(param1, param2):
    # 你的Spectral oASIS特征选择逻辑
    # ...
    return result

if __name__ == '__main__':
    with multiprocessing.Pool() as pool:
        params = [(p1_1, p2_1), (p1_2, p2_2), ...]
        results = pool.starmap(run_SpectraloASIS, params)

注意：spawn方式会重新导入模块，需确保run_SpectraloASIS定义在if __name__ == '__main__':块之外。

内容的提问来源于stack exchange，提问作者DaPi123