Python多进程嵌套Pool：外层Pool返回生成器而非结果的问题

你的问题核心是ProcessPoolExecutor.map返回的是惰性迭代器，并非立即生成的列表，直接打印它只会输出迭代器对象，而非实际处理结果。同时因为惰性求值特性，程序会先打印迭代器信息，再开始执行任务。

修复代码 #

只需要把外层map返回的迭代器转换成列表，强制等待所有任务完成并收集结果：

from concurrent.futures import ProcessPoolExecutor as Pool

num_pool = 10


def main_pool(num):
    numbers = []
    print('MAIN POOL', num)
    strings_write = (f'{num}-{i}' for i in range(num))
    with Pool(num) as subp:
        numbers.extend(subp.map(sub_pool, strings_write))
    return numbers


def sub_pool(x):
    print('sub_pool', x)
    print(f'{x}')
    return x


if __name__ == "__main__":
    with Pool(num_pool) as p:
        # 将map返回的迭代器转为列表，触发结果收集
        result = list(p.map(main_pool, list(range(1, num_pool + 1))))
        print('Result Outside')
        print(result)

效果说明 #

修改后程序会先等待所有外层、内层进程池任务执行完毕，再打印嵌套结构的最终结果，格式类似：

[['1-0'], ['2-0', '2-1'], ['3-0', '3-1', '3-2'], ..., ['10-0', '10-1', ..., '10-9']]

如果需要扁平化的一维结果，可以借助itertools.chain合并：

from concurrent.futures import ProcessPoolExecutor as Pool
from itertools import chain

num_pool = 10

# 其余函数代码不变

if __name__ == "__main__":
    with Pool(num_pool) as p:
        result = list(chain.from_iterable(p.map(main_pool, list(range(1, num_pool + 1)))))
        print('Result Outside')
        print(result)

最终会得到所有子任务返回值的一维列表：

['1-0', '2-0', '2-1', '3-0', '3-1', '3-2', ..., '10-9']

内容的提问来源于stack exchange，提问作者Volatil3