Python Multiprocessing Pool内部模块报错：缺失doc和pos参数求助

错误信息 #

Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\threading.py", line 950, in _bootstrap_inner
    self.run()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\threading.py", line 888, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py", line 576, in _handle_results
    task = get()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py", line 256, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\exceptions.py", line 41, in __init__
    CompatJSONDecodeError.__init__(self, *args)
TypeError: __init__() missing 2 required positional arguments: 'doc' and 'pos'

错误成因 #

这个问题本质是多进程间的对象序列化/反序列化（pickle）失败：

• requests库的CompatJSONDecodeError异常类初始化时需要doc和pos两个参数，但当子进程抛出这个异常后，multiprocessing用pickle传递异常对象回主进程时，没有正确保留这两个参数，导致反序列化时无法初始化异常类，触发报错。
• 场景通常是子进程执行requests请求时遇到JSON解码错误，异常对象跨进程传递时出了问题。

解决方法 #

1. 捕获异常并转为可序列化类型 #

在子进程的执行函数里，捕获requests相关的JSON解码异常，不要直接抛出原异常，而是将异常信息转为字符串或自定义的可序列化对象后再抛出：

import requests
from multiprocessing import Pool

def worker_func(url):
    try:
        resp = requests.get(url)
        resp.json()  # 可能触发JSONDecodeError
    except requests.exceptions.JSONDecodeError as e:
        # 转为RuntimeError，避免传递原异常对象
        raise RuntimeError(f"JSON解码失败: {str(e)}") from e
    # 其他业务逻辑...

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(worker_func, ["https://example.com/invalid-json"])

2. 升级requests库到兼容版本 #

旧版本requests的CompatJSONDecodeError存在pickle兼容性问题，升级到2.28.0及以上版本大概率能解决：

pip install --upgrade requests

3. 重构代码，避免传递异常对象 #

让子进程返回状态+结果的元组，主进程根据状态处理，完全绕开异常跨进程传递的问题：

import requests
from multiprocessing import Pool

def worker_func(url):
    try:
        resp = requests.get(url)
        data = resp.json()
        return (True, data)
    except Exception as e:
        return (False, f"执行失败: {str(e)}")

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(worker_func, ["https://example.com/invalid-json"])
        for success, content in results:
            if success:
                print("获取结果:", content)
            else:
                print("错误信息:", content)

内容的提问来源于stack exchange，提问作者8bitdefault