为何aiohttp异步请求与requests同步请求耗时相近？

我有227个需要获取HTML内容的链接，使用Python与Jupyter Notebook开发。发现用aiohttp和asyncio实现的异步请求，耗时与requests库的同步请求几乎一致。

原异步请求代码 #

import asyncio
import aiohttp

# 在Jupyter中需初始化nest_asyncio
import nest_asyncio
nest_asyncio.apply()

all_links = [] # 存储所有目标链接
htmls = []

try:
       async def main():
              try:
                 async with aiohttp.ClientSession() as session:
                     for link in all_links:
                            async with session.get(link) as response:
                                   html = await response.text()
                                   htmls.append(html)
              except Exception as e:
                       print(e)
       loop = asyncio.get_event_loop()
       loop.run_until_complete(main())
except Exception as e:
       print(e)

# 耗时：1分9.2秒完成227次请求

原同步请求代码 #

import requests

for link in all_links:
     value = requests.get(link)

# 耗时：1分9.5秒完成227次请求

测试链接（部分） #

• https://www.eslcafe.com/postajob-detail/great-private-and-public-school-positions-all?koreasearch=&koreapageno=1&koreapagesize=300&chinasearch=&chinapageno=1&chinapagesize=300&internationalsearch=&internationalpageno=1&internationalpagesize=300
• https://www.eslcafe.com/postajob-detail/apply-now-for-teaching-job-in-korea-china-hig-14?koreasearch=&koreapageno=1&koreapagesize=300&chinasearch=&chinapageno=1&chinapagesize=300&internationalsearch=&internationalpageno=1&internationalpagesize=300
• https://www.eslcafe.com/postajob-detail/full-time-esl-teacher-wanted-2?koreasearch=&koreapageno=1&koreapagesize=300&chinasearch=&chinapageno=1&chinapagesize=300&internationalsearch=&internationalpageno=1&internationalpagesize=300
• https://www.eslcafe.com/postajob-detail/teachers-for-march-seoul-good-locations-acros?koreasearch=&koreapageno=1&koreapagesize=300&chinasearch=&chinapageno=1&chinapagesize=300&internationalsearch=&internationalpageno=1&internationalpagesize=300
• https://www.eslcafe.com/postajob-detail/asap-2023-fall-semester-open-full-time-trustw-4?koreasearch=&koreapageno=1&koreapagesize=300&chinasearch=&chinapageno=1&chinapagesize=300&internationalsearch=&internationalpageno=1&internationalpagesize=300

问题根源 #

你的异步代码本质是串行执行，没有实现真正的并发。在for link in all_links循环中，每次都await session.get(link)，必须等当前请求完全结束才会发起下一个请求，和同步循环的执行逻辑没有区别，自然耗时相近。

修复后的异步代码 #

import asyncio
import aiohttp
import nest_asyncio

nest_asyncio.apply()

all_links = [
    # 填入你的所有目标链接
    "https://www.eslcafe.com/postajob-detail/great-private-and-public-school-positions-all?koreasearch=&koreapageno=1&koreapagesize=300&chinasearch=&chinapageno=1&chinapagesize=300&internationalsearch=&internationalpageno=1&internationalpagesize=300",
    "https://www.eslcafe.com/postajob-detail/apply-now-for-teaching-job-in-korea-china-hig-14?koreasearch=&koreapageno=1&koreapagesize=300&chinasearch=&chinapageno=1&chinapagesize=300&internationalsearch=&internationalpageno=1&internationalpagesize=300",
    # 其他链接...
]
htmls = []

async def fetch_html(session, link):
    """单个链接的HTML获取逻辑"""
    try:
        async with session.get(link) as response:
            return await response.text()
    except Exception as e:
        print(f"请求{link}失败: {str(e)}")
        return None

async def main():
    async with aiohttp.ClientSession() as session:
        # 创建所有请求任务
        tasks = [fetch_html(session, link) for link in all_links]
        # 并发执行所有任务并收集结果
        results = await asyncio.gather(*tasks)
        # 过滤掉失败的请求结果
        htmls.extend([res for res in results if res is not None])

# 运行异步主函数
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

关键改进点 #

1. 拆分请求逻辑：把单个链接的请求封装成独立的fetch_html函数，每个请求对应一个异步任务。
2. 并发执行任务：通过asyncio.gather(*tasks)一次性发起所有请求，等待所有任务完成后统一收集结果，真正利用异步IO的并发特性。
3. 错误处理细化：针对单个请求做错误捕获，避免一个请求失败导致整个任务中断。

额外优化建议 #

• 调整连接池大小：aiohttp默认限制100个并发连接，若需要更高并发，可在创建ClientSession时设置：

  connector = aiohttp.TCPConnector(limit=200)  # 根据服务器承受能力调整数值
  async with aiohttp.ClientSession(connector=connector) as session:
  

• 加入超时设置：避免单个请求卡住影响整体，可在session.get中添加超时参数：

  async with session.get(link, timeout=aiohttp.ClientTimeout(total=10)) as response:
  

内容的提问来源于stack exchange，提问作者MARUF