为何asyncio.create_task与ensure_future创建httpx任务用于gather时表现不同？

问题重现 #

运行异步httpx请求示例时，使用asyncio.ensure_future可正常获取所有宝可梦数据，但替换为官方推荐的asyncio.create_task后，当请求数量达到150个时会出现长时间等待并抛出DNS解析异常，报错核心信息如下：

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/anyio/_core/_sockets.py", line 186, in connect_tcp
    addr_obj = ip_address(remote_host)
  File "/usr/lib/python3.10/ipaddress.py", line 54, in ip_address
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
ValueError: 'pokeapi.co' does not appear to be an IPv4 or IPv6 address

缩小请求数量（如范围改为1-70）则问题消失。使用环境为Python 3.10.5 + httpx 0.23.0，替换后的核心代码如下：

import asyncio
import httpx
import time

start_time = time.time()

async def get_pokemon(client, url):
        resp = await client.get(url)
        pokemon = resp.json()
        return pokemon['name']
    
async def main():
    async with httpx.AsyncClient() as client:
        tasks = []
        for number in range(1, 151):
            url = f'https://pokeapi.co/api/v2/pokemon/{number}'
            tasks.append(asyncio.create_task(get_pokemon(client, url)))  # 替换为create_task的代码
        original_pokemon = await asyncio.gather(*tasks)
        for pokemon in original_pokemon:
            print(pokemon)

asyncio.run(main())
print("--- %s seconds ---" % (time.time() - start_time))

原因分析 #

1. 任务调度时机差异：虽然ensure_future和create_task核心功能都是创建异步任务，但create_task会立即将任务加入当前事件循环的待执行队列，而ensure_future在批量创建任务时调度相对平缓。当同时创建150个任务时，create_task会瞬间触发大量并发DNS查询，超出了旧版本httpx底层解析逻辑的处理能力。
2. httpx旧版本DNS并发bug：这是httpx 0.23.x版本依赖的anyio库的已知缺陷，高并发场景下DNS解析逻辑会出现异常，错误地将域名直接当作IP地址解析，从而抛出ValueError。

解决办法 #

• 升级httpx到最新稳定版本：httpx 0.24.0及后续版本已修复该DNS并发问题，执行以下命令即可升级：

  pip install --upgrade httpx
  

• 限制并发请求数量：通过asyncio.Semaphore控制同时发起的请求数，避免瞬间触发过高的DNS查询压力，修改后的示例代码如下：

  import asyncio
  import httpx
  import time
  
  start_time = time.time()
  semaphore = asyncio.Semaphore(50)  # 限制同时最多50个并发请求
  
  async def get_pokemon(client, url):
      async with semaphore:
          resp = await client.get(url)
          pokemon = resp.json()
          return pokemon['name']
      
  async def main():
      async with httpx.AsyncClient() as client:
          tasks = []
          for number in range(1, 151):
              url = f'https://pokeapi.co/api/v2/pokemon/{number}'
              tasks.append(asyncio.create_task(get_pokemon(client, url)))
          original_pokemon = await asyncio.gather(*tasks)
          for pokemon in original_pokemon:
              print(pokemon)
  
  asyncio.run(main())
  print("--- %s seconds ---" % (time.time() - start_time))
  

内容的提问来源于stack exchange，提问作者Lesser Cormorant