Python concurrent.futures: ProcessPoolExecutor报BrokenProcessPool错误

以下Python代码在Windows 10系统运行时触发BrokenProcessPool错误，代码8个月前可正常运行，目前在Jupyter Notebook和Spyder中均报错，但Ubuntu终端可正常执行，已确认包含__name__=="__main__"语句。

出错代码 #

import random, matplotlib.pyplot as plt, time, math, multiprocessing, concurrent.futures
import numpy as np

def flatten(xss):
    return [x for xs in xss for x in xs]

ram0=1024

def sticky_pos(iterations):
    # do something
    return val

def main():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        start_time = time.perf_counter()
        temp = list(executor.map(sticky_pos, [10 for i in range(int(iterations/10))]))  #int(iterations/10)
        finish_time = time.perf_counter()
    print(f"Program finished in {finish_time-start_time} seconds")

if __name__ == "__main__":
    main()

pos=flatten(temp)

报错信息 #

---------------------------------------------------------------------------
BrokenProcessPool                         Traceback (most recent call last)
<ipython-input-1-dec2d19e0a0b> in <module>
     81 
     82 if __name__ == "__main__":
---&gt; 83     main()
     84 
     85 pos=flatten(temp)

<ipython-input-1-dec2d19e0a0b> in main()
     76     with concurrent.futures.ProcessPoolExecutor() as executor:
     77         start_time = time.perf_counter()
---&gt; 78         temp = list(executor.map(sticky_pos, [10 for i in range(int(iterations/10))]))  #int(iterations/10)
     79         finish_time = time.perf_counter()
     80     print(f"Program finished in {finish_time-start_time} seconds")

~\AppData\Local\Continuum\anaconda3\lib\concurrent\futures\process.py in _chain_from_iterable_of_lists(iterable)
    481     careful not to keep references to yielded objects.
    482     """
--&gt; 483     for element in iterable:
    484         element.reverse()
    485         while element:

~\AppData\Local\Continuum\anaconda3\lib\concurrent\futures\_base.py in result_iterator()
    596                     # Careful not to keep a reference to the popped future
    597                     if timeout is None:
--&gt; 598                         yield fs.pop().result()
    599                     else:
    600                         yield fs.pop().result(end_time - time.monotonic())

~\AppData\Local\Continuum\anaconda3\lib\concurrent\futures\_base.py in result(self, timeout)
    433                 raise CancelledError()
    434             elif self._state == FINISHED:
--&gt; 435                 return self.__get_result()
    436             else:
    437                 raise TimeoutError()

~\AppData\Local\Continuum\anaconda3\lib\concurrent\futures\_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--&gt; 384             raise self._exception
    385         else:
    386             return self._result

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

解决方法 #

• 修复全局代码执行问题：Windows下多进程会重新导入主模块，pos=flatten(temp)写在if __name__ == "__main__"块外，会被子进程执行，但temp仅在main()内定义，子进程运行时temp未初始化，会抛出异常导致进程崩溃。将该语句移到main()函数内部：

  def main():
      with concurrent.futures.ProcessPoolExecutor() as executor:
          start_time = time.perf_counter()
          temp = list(executor.map(sticky_pos, [10 for i in range(int(iterations/10))]))
          finish_time = time.perf_counter()
      print(f"Program finished in {finish_time-start_time} seconds")
      pos = flatten(temp)
      # 后续对pos的操作也放在此处
  

• 检查sticky_pos函数逻辑：函数内的业务代码可能包含Windows不兼容的操作（如文件路径格式、系统调用），或内存占用过高导致进程被系统终止。可在函数内添加打印/日志语句，定位具体崩溃点；若涉及内存操作，排查是否存在内存泄漏或超出系统资源限制。
• 限制进程池大小：Windows默认进程池大小为CPU核心数，核心数较多时，并发进程可能占用过多资源导致崩溃。显式指定进程数：

  with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor:
      # 后续代码不变
  

• 更新Python环境：Anaconda环境可能因依赖更新产生冲突，尝试更新相关依赖包，或创建新虚拟环境重新安装所需库。
• 避免交互式环境运行：Jupyter Notebook和Spyder这类交互式环境对多进程支持存在兼容性问题，尽量在Windows命令行（cmd/PowerShell）中直接运行脚本。

内容的提问来源于stack exchange，提问作者Razor