You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

Python异步程序读取响应HTML时冻结的问题排查求助

Python异步程序读取响应HTML时冻结的问题排查求助

我最近开始学习Python的asyncio,在练习队列的时候遇到了问题。我的任务是写一段代码,从网站获取随机图片,解析图片链接,并用异步队列下载这些图片。但不幸的是,我的程序在尝试读取响应HTML的时候直接卡住了,完全不知道该怎么解决。

以下是我的代码:

import asyncio
import aiohttp
from bs4 import BeautifulSoup
from concurrent.futures import ProcessPoolExecutor

async def make_request(url, session):
    async with session.get(url) as response:
        if response.ok:
            return response
        else:
            print(f'{url} returned: {response.status}')


async def get_image_page(queue, session):
    url = "https://c.xkcd.com/random/comic/"
    response = await make_request(url, session)
    await queue.put(str(response.url))

def parse_link(html):
    soup = BeautifulSoup(html, 'lxml')
    u = soup.select_one('div#comic>img').get('src')
    image_url = f'https:{u}'
    return image_url


async def get_image_url(pages_queue, image_urls_queue, session):
    while True:
        url = await pages_queue.get()
        response = await make_request(url, session)
        html = await response.text()

        loop = asyncio.get_running_loop()
        with ProcessPoolExecutor() as pool:
            image_url = await loop.run_in_executor(
                pool, parse_link, html
            )

        await image_urls_queue.put(image_url)

        pages_queue.task_done()


async def download_image(url):
    pass

async def main():
    async with aiohttp.ClientSession() as session:
        pages_queue = asyncio.Queue()
        image_urls_queue = asyncio.Queue()

        page_getters = []
        for _ in range(4):
            task = asyncio.create_task(
                get_image_page(pages_queue, session)
            )
            page_getters.append(task)
        
        url_getters = []
        for _ in range(4):
            task = asyncio.create_task(
                get_image_url(pages_queue, image_urls_queue, session)
            )
            url_getters.append(task)

        await asyncio.gather(*page_getters)
        await pages_queue.join()

        for task in page_getters:
            task.cancel()
        
        print(image_urls_queue)


if __name__ == '__main__':
    asyncio.run(main())

我尝试调试get_image_url()协程,发现问题出在这一行:

html = await response.text()

有没有人能指出我哪里做错了,以及该怎么修复?

备注:内容来源于stack exchange,提问作者Mushinto

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.04.14 09:19:35