使用asyncio并行运行协程却同步执行,如何实现真正并行?
问题分析
asyncio 是单线程异步框架,协程切换的核心前提是当前协程遇到可等待的异步操作并执行await,此时事件循环才会主动切换到其他协程。但你的func和func1是纯CPU密集的同步函数,执行时会持续占用线程资源,不会触发协程切换,导致两个协程只能串行执行,总耗时为两个任务耗时之和。
解决方案
要实现真正的并行执行,需要把阻塞的CPU密集任务从asyncio的单线程事件循环中剥离,放到线程池或进程池中执行,让事件循环可以在等待任务完成时切换到其他协程。
方案1:使用asyncio.to_thread(Python 3.9+)
将同步函数放到线程池执行,代码简洁,适合轻度CPU密集或IO密集任务:
import asyncio import time def func(): print("into the func") s = time.time() l = [] for i in range(100000000): l.append(i) print("time in func is ", time.time() - s) def func1(): print("into the func1") s = time.time() l = [] for i in range(100000000): l.append(i) print("time in func1 is ", time.time() - s) async def exp(): print("exp started") await asyncio.sleep(0.01) # 把同步函数提交到线程池执行,await等待结果 await asyncio.to_thread(func) print("finished exp") return [1,2] async def data(): print("data started") await asyncio.sleep(0.01) await asyncio.to_thread(func1) print("finished data") return {'a':1} async def start_coros(): tasks = [asyncio.create_task(coro) for coro in (exp(), data())] await asyncio.wait(tasks) # 统计总耗时 start_total = time.time() asyncio.run(start_coros()) print("total time :", time.time() - start_total)
方案2:使用concurrent.futures.ProcessPoolExecutor
由于Python的GIL(全局解释器锁)限制了单线程CPU利用率,重度CPU密集任务更适合用进程池,能充分利用多核CPU,总耗时更接近单个任务的执行时间:
import asyncio import time from concurrent.futures import ProcessPoolExecutor def func(): print("into the func") s = time.time() l = [] for i in range(100000000): l.append(i) print("time in func is ", time.time() - s) def func1(): print("into the func1") s = time.time() l = [] for i in range(100000000): l.append(i) print("time in func1 is ", time.time() - s) async def exp(executor): print("exp started") await asyncio.sleep(0.01) # 提交任务到进程池执行 await asyncio.get_event_loop().run_in_executor(executor, func) print("finished exp") return [1,2] async def data(executor): print("data started") await asyncio.sleep(0.01) await asyncio.get_event_loop().run_in_executor(executor, func1) print("finished data") return {'a':1} async def start_coros(): # 创建进程池,max_workers建议设为CPU核心数 with ProcessPoolExecutor(max_workers=2) as executor: tasks = [ asyncio.create_task(exp(executor)), asyncio.create_task(data(executor)) ] await asyncio.wait(tasks) start_total = time.time() asyncio.run(start_coros()) print("total time :", time.time() - start_total)
内容的提问来源于stack exchange,提问作者coder
相关产品推荐
相关产品推荐

