Python多进程中调用list()处理迭代器后无报错的原因分析

复现代码 #

from itertools import repeat
import multiprocessing

# from stackexchange
def starmap_with_kwargs(pool, fn, args_iter, kwargs_iter):
    args_for_starmap = zip(repeat(fn), args_iter, kwargs_iter)
    print(args_iter)
    return pool.starmap(apply_args_and_kwargs, args_for_starmap)

def apply_args_and_kwargs(fn, args, kwargs):
    print('test')
    return fn(*args, **kwargs)

def func(path, dictArg, **kwargs):
    for i in dictArg:
        print(i['a'])
        print(kwargs['yes'])

def funcWrapper(path, dictList, **kwargs):

    args_iter = zip(repeat(path), dictList)
    kwargs_iter = repeat(kwargs)

    # list(args_iter)

    pool = multiprocessing.Pool()
    starmap_with_kwargs(pool, func, args_iter, kwargs_iter)
       
    
dictList = [{'a: 2'}, {'a': 65}, {'a': 213}, {'a': 3218}]
path = 'some/path/to/something'

funcWrapper(path, dictList, yes=1)

问题详情 #

运行上述代码时，会出现预期的TypeError错误（该错误可通过移除func中的循环修复）；但如果取消注释list(args_iter)这一行，就不会出现任何错误信息。想知道为什么添加list(args_iter)后就没有报错了？

使用环境：WSL中的Ubuntu 20.04.6 LTS，Python版本为3.8.10。

移除list(args_iter)时的错误信息 #

<zip object at 0x7fa1ec0b8340>
test
test
test
test
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "testing.py", line 67, in apply_args_and_kwargs
    return fn(*args, **kwargs)
  File "testing.py", line 71, in func
    print(i['a'])
TypeError: string indices must be integers
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "testing.py", line 88, in <module>
    funcWrapper(path, dictList, yes=1)
  File "testing.py", line 82, in funcWrapper
    starmap_with_kwargs(pool, func, args_iter, kwargs_iter)
  File "testing.py", line 61, in starmap_with_kwargs
    return pool.starmap(apply_args_and_kwargs, args_for_starmap)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
TypeError: string indices must be integers

原因解析 #

这是迭代器一次性特性和多进程任务执行逻辑共同导致的：

• 迭代器只能遍历一次：zip(repeat(path), dictList)返回的是迭代器，调用list(args_iter)会一次性取出迭代器里的所有元素，此时原args_iter已经被耗尽，没有任何数据剩余。
• 多进程池无任务可执行：当把耗尽的迭代器传给pool.starmap时，因为迭代器里没有元素，多进程池不会执行任何func任务，自然不会触发func里的错误逻辑——也就是遍历到dictList中第一个错误元素{'a: 2'}（这是个含字符串的集合，不是字典，访问i['a']会触发TypeError）。
• 额外补充：你代码里dictList的第一个元素存在语法错误，正确写法应为{'a': 2}，这才是最初报错的根本原因。当迭代器被耗尽后，任务未执行，这个错误就不会暴露出来。

内容的提问来源于stack exchange，提问作者Iram Haque