Jupyter Notebook中ProcessPoolExecutor并行执行失败问题及解决

问题重现 #

在Jupyter Notebook中搭建测试并行执行的简单环境时，ProcessPoolExecutor始终执行失败。设备可用核心远超4个，且无内存问题，但所有任务均报错。

示例代码：

from concurrent.futures import ProcessPoolExecutor

def worker(i):
    print(f"TASK: {i}")
    result = i * i
    print(f"RESULT: {result}")
    return result


futures = []

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=4) as executor:
        for i in range(10):
            futures.append(executor.submit(worker, i))

    for i, future in enumerate(futures):
        try:
            result = future.result()
            print(result)
        except Exception as e:
            print(f"Error in future {i}: {e}")

报错信息：

Error in future 0: A process in the process pool was terminated abruptly while the future was running or pending.
(and so on for the other futures)

解决方案 #

将worker函数单独保存到一个Python文件中，在Jupyter Notebook里导入该函数后重新执行，即可正常运行。

内容的提问来源于stack exchange，提问作者Alex F