使用forkserver时无法传递multiprocessing.Manager：无法pickle弱引用对象

当使用forkserver作为Python多进程启动方式时，直接把multiprocessing.Manager实例传给子进程，会触发以下错误：

TypeError: cannot pickle 'weakref.ReferenceType' object

示例代码 #

import multiprocessing

def main():
    multiprocessing.set_start_method('forkserver')
    with multiprocessing.Manager() as manager:
        proc = multiprocessing.Process(target=fun, args=('p1', manager))
        proc.start()
        proc.join()

def fun(name, manager):
    print('fun:', name, manager)

if __name__ == '__main__':
    main()

报错详情 #

$ /opt/python3.12.6/bin/python3 ./demo_mp_manager_forkserver.py
Traceback (most recent call last):
  File "/home/.../demo_mp_manager_forkserver.py", line 14, in <module>
    main()
  File "/home/.../demo_mp_manager_forkserver.py", line 7, in main
    proc.start()
  File "/opt/python3.12.6/lib/python3.12/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/opt/python3.12.6/lib/python3.12/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python3.12.6/lib/python3.12/multiprocessing/context.py", line 301, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/python3.12.6/lib/python3.12/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
  File "/opt/python3.12.6/lib/python3.12/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/python3.12.6/lib/python3.12/multiprocessing/popen_forkserver.py", line 47, in _launch
    reduction.dump(process_obj, buf)
  File "/opt/python3.12.6/lib/python3.12/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'weakref.ReferenceType' object

原因分析 #

multiprocessing.Manager内部包含弱引用对象，这类对象无法被Python的ForkingPickler序列化。而forkserver启动方式要求所有传递给子进程的参数必须先序列化，再传给forkserver进程去创建子进程，所以直接传递Manager实例会失败。

解决方案 #

1. 传递Manager创建的共享对象（推荐）

不要直接传Manager实例，而是传它创建的共享代理对象（比如manager.dict()、manager.list()等），这些代理对象支持序列化，可以安全传递给子进程。

修改后的代码：

import multiprocessing

def main():
    multiprocessing.set_start_method('forkserver')
    with multiprocessing.Manager() as manager:
        # 创建共享字典
        shared_dict = manager.dict()
        proc = multiprocessing.Process(target=fun, args=('p1', shared_dict))
        proc.start()
        proc.join()
        print('主进程:', shared_dict)

def fun(name, shared_dict):
    shared_dict['name'] = name
    print('子进程:', name, shared_dict)

if __name__ == '__main__':
    main()

2. 子进程中重新连接到Manager

如果需要在子进程中使用Manager的其他功能（比如动态创建新的共享对象），可以获取Manager的地址，在子进程中重新连接到这个Manager。

修改后的代码：

import multiprocessing
from multiprocessing.managers import SyncManager

def main():
    multiprocessing.set_start_method('forkserver')
    with multiprocessing.Manager() as manager:
        # 获取Manager的地址信息
        manager_addr = manager._address
        proc = multiprocessing.Process(target=fun, args=('p1', manager_addr))
        proc.start()
        proc.join()

def fun(name, manager_addr):
    # 重新连接到已运行的Manager
    manager = SyncManager(address=manager_addr)
    manager.connect()
    # 创建新的共享列表
    shared_list = manager.list([1, 2, 3])
    shared_list.append(name)
    print('子进程:', name, shared_list)

if __name__ == '__main__':
    main()

内容的提问来源于stack exchange，提问作者Messa