为何Python HTTPX客户端创建耗时过长？求批量创建优化方案

问题核心原因 #

每个httpx.AsyncClient默认会独立初始化一套HTTP传输层组件（包括SSL上下文、连接池等），这些初始化操作本身属于CPU密集型且耗时的步骤，当创建数千个实例时，重复的初始化开销会被急剧放大。

解决方案1：共享底层传输层（Transport） #

创建一个全局的AsyncHTTPTransport实例，让所有AsyncClient复用该传输层，避免重复执行SSL上下文和连接池的初始化操作。

import httpx
import time
import asyncio

async def main():
    total_clients = 100
    # 创建共享的传输层实例
    transport = httpx.AsyncHTTPTransport()
    
    start_time = time.time()
    # 所有客户端复用同一个transport
    clients = [httpx.AsyncClient(transport=transport) for _ in range(total_clients)]
    end_time = time.time()
    
    print(f'{total_clients} httpx clients were created in {(end_time - start_time):.2f} seconds.')
    
    # 统一关闭资源
    for client in clients:
        await client.aclose()
    await transport.aclose()

if __name__ == '__main__':
    asyncio.run(main())

解决方案2：基于基础客户端克隆（copy） #

如果每个客户端需要独立的Cookies、Headers或Proxy配置，可以先创建一个基础客户端，再通过copy()方法克隆新实例。克隆操作会复用底层传输层，耗时几乎可以忽略。

import httpx
import time
import asyncio

async def main():
    total_clients = 100
    # 创建基础客户端
    base_client = httpx.AsyncClient()
    
    start_time = time.time()
    clients = []
    for i in range(total_clients):
        # 克隆并设置独立配置
        client = base_client.copy(
            headers={"X-Unique-ID": str(i)},
            cookies={"session_id": f"sid_{i}"},
            # 可添加proxies等其他差异化配置
        )
        clients.append(client)
    end_time = time.time()
    
    print(f'{total_clients} httpx clients were created in {(end_time - start_time):.2f} seconds.')
    
    # 关闭所有资源
    for client in clients:
        await client.aclose()
    await base_client.aclose()

if __name__ == '__main__':
    asyncio.run(main())

内容的提问来源于stack exchange，提问作者davi-ximenes