如何使用Gevent并行执行带参任务并获取返回结果?
问题描述
我有一段基于gevent的并行代码,原本能正常运行并打印计算结果,耗时约1秒:
from gevent import monkey; monkey.patch_all() from gevent.pool import Pool import time def work(x): time.sleep(1) # 可替换为IO相关操作,比如requests.get() newVal = x ** 2 print(x, newVal) start = time.perf_counter() p = Pool() l = [x for x in range(10)] for i,x in enumerate(l): p.spawn(work, x) p.join() print(round(time.perf_counter() - start, 3)) # 输出约1.001或1.002
运行输出:
(test) C:\DEV\Python-Diverses\gevent>python try.py 0 0 1 1 2 4 3 9 4 16 5 25 6 36 7 49 8 64 9 81 1.036
我希望从work函数返回计算结果并收集所有值,于是修改了代码:
from gevent import monkey; monkey.patch_all() from gevent.pool import Pool import time def work(x): time.sleep(1) # 可替换为IO相关操作,比如requests.get() newVal = x ** 2 return (newVal) start = time.perf_counter() p = Pool() l = [x for x in range(10)] ergList = [] for i,x in enumerate(l): erg = p.spawn(work, x) ergList.append(erg) p.join() print(ergList) print(round(time.perf_counter() - start, 3)) # 输出约1.001或1.002
运行耗时仍为1秒,但输出的是Greenlet实例而非计算结果:
(test) C:\DEV\Python-Diverses\gevent>python try.py [<Greenlet at 0x180704a55e0: _run>, <Greenlet at 0x18070533f40: _run>, <Greenlet at 0x18070563180: _run>, <Greenlet at 0x180705632c0: _run>, <Greenlet at 0x18070563400: _run>, <Greenlet at 0x18070563540: _run>, <Greenlet at 0x18070563680: _run>, <Greenlet at 0x180705637c0: _run>, <Greenlet at 0x18070563900: _run>, <Greenlet at 0x18070563a40: _run>] 1.07
请问如何在并行执行任务的同时获取返回的计算结果,而非Greenlet实例?
解决方案
方法1:调用Greenlet实例的get()方法
p.spawn()返回的是Greenlet对象,调用它的get()方法可以获取函数的返回值。需在p.join()之后调用,确保所有任务执行完毕:
from gevent import monkey; monkey.patch_all() from gevent.pool import Pool import time def work(x): time.sleep(1) newVal = x ** 2 return newVal start = time.perf_counter() p = Pool() l = [x for x in range(10)] ergList = [] for x in l: erg = p.spawn(work, x) ergList.append(erg) p.join() # 遍历Greenlet列表,提取返回结果 results = [g.get() for g in ergList] print(results) print(round(time.perf_counter() - start, 3))
运行输出:
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81] 1.05
方法2:使用Pool.map()简化代码
gevent的Pool提供了map()方法,与Python内置map功能类似,但会并行执行任务,直接返回结果列表,无需手动处理Greenlet对象:
from gevent import monkey; monkey.patch_all() from gevent.pool import Pool import time def work(x): time.sleep(1) newVal = x ** 2 return newVal start = time.perf_counter() p = Pool() l = [x for x in range(10)] # 并行执行任务并直接获取结果 results = p.map(work, l) print(results) print(round(time.perf_counter() - start, 3))
运行输出与方法1一致,代码更简洁直观。
内容的提问来源于stack exchange,提问作者Rapid1898
相关产品推荐
相关产品推荐

