Python异步程序读取响应HTML时冻结的问题排查求助
Python异步程序读取响应HTML时冻结的问题排查求助
我最近开始学习Python的asyncio,在练习队列的时候遇到了问题。我的任务是写一段代码,从网站获取随机图片,解析图片链接,并用异步队列下载这些图片。但不幸的是,我的程序在尝试读取响应HTML的时候直接卡住了,完全不知道该怎么解决。
以下是我的代码:
import asyncio import aiohttp from bs4 import BeautifulSoup from concurrent.futures import ProcessPoolExecutor async def make_request(url, session): async with session.get(url) as response: if response.ok: return response else: print(f'{url} returned: {response.status}') async def get_image_page(queue, session): url = "https://c.xkcd.com/random/comic/" response = await make_request(url, session) await queue.put(str(response.url)) def parse_link(html): soup = BeautifulSoup(html, 'lxml') u = soup.select_one('div#comic>img').get('src') image_url = f'https:{u}' return image_url async def get_image_url(pages_queue, image_urls_queue, session): while True: url = await pages_queue.get() response = await make_request(url, session) html = await response.text() loop = asyncio.get_running_loop() with ProcessPoolExecutor() as pool: image_url = await loop.run_in_executor( pool, parse_link, html ) await image_urls_queue.put(image_url) pages_queue.task_done() async def download_image(url): pass async def main(): async with aiohttp.ClientSession() as session: pages_queue = asyncio.Queue() image_urls_queue = asyncio.Queue() page_getters = [] for _ in range(4): task = asyncio.create_task( get_image_page(pages_queue, session) ) page_getters.append(task) url_getters = [] for _ in range(4): task = asyncio.create_task( get_image_url(pages_queue, image_urls_queue, session) ) url_getters.append(task) await asyncio.gather(*page_getters) await pages_queue.join() for task in page_getters: task.cancel() print(image_urls_queue) if __name__ == '__main__': asyncio.run(main())
我尝试调试get_image_url()协程,发现问题出在这一行:
html = await response.text()
有没有人能指出我哪里做错了,以及该怎么修复?
备注:内容来源于stack exchange,提问作者Mushinto
相关产品推荐
相关产品推荐

