Multiprocessing执行异常：返回值与共享字典均为空求助

原代码 #

import multiprocessing as mp
from multiprocessing.managers import SyncManager

n_cores = mp.cpu_count()

def parallel_fn(job_n, cache):
    cache['job_b'] = job_n
    return job_n

if __name__=="__main__":
    with SyncManager() as manager:

        shared_cache = manager.dict()
        
        args = list(zip(range(n_cores), shared_cache))

        with mp.Pool(n_cores) as pool:

            result = pool.starmap(parallel_fn, args)

            print(result)
        
        print(shared_cache)

运行输出 #

16
Shared dict before: {}
Pool return: []
Shared dict after: {}

问题原因 #

1. 任务参数列表为空：zip(range(n_cores), shared_cache)中，shared_cache是空字典，zip会以最短可迭代对象的长度为准，因此生成的args是空列表，导致进程池没有任务可执行，返回空结果。
2. 共享字典键冲突：即使参数正确，所有进程都对cache['job_b']赋值，后执行的进程会覆盖先执行进程的结果，最终只能保留最后一个进程的赋值。

修正后的代码 #

import multiprocessing as mp
from multiprocessing.managers import SyncManager

n_cores = mp.cpu_count()

def parallel_fn(job_n, cache):
    # 使用唯一key避免覆盖
    cache[f'job_{job_n}'] = job_n
    return job_n

if __name__=="__main__":
    with SyncManager() as manager:
        shared_cache = manager.dict()
        # 正确生成每个任务的参数：每个任务传递job编号和共享字典
        args = [(job_n, shared_cache) for job_n in range(n_cores)]
        
        with mp.Pool(n_cores) as pool:
            result = pool.starmap(parallel_fn, args)
            print("Pool return:", result)
        
        print("Shared dict after:", shared_cache)

预期输出 #

Pool return: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Shared dict after: {'job_0': 0, 'job_1': 1, ..., 'job_15': 15}

内容的提问来源于stack exchange，提问作者Andrea