httpx内存未释放问题排查求助

我正试图找到无需替换httpx的解决方案，毕竟支持HTTP2的异步库十分有限。在等待httpx团队回复期间，我希望确认当前遇到的情况是库的潜在问题还是自身经验不足。

测试代码 #

import httpx
import asyncio
from memory_profiler import profile
import aiohttp


@profile(precision=4)
async def memory_test(url):
    
    '''
        async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            html = await response.text()    
            print(f'Length of response is: {len(html)}')
    '''
    async with httpx.AsyncClient(http2=True) as client:
        
        html = await client.get(url, follow_redirects=True)
        print(f'Length of response is: {len(html.text)}')   
        
    del html    
    return None

async def main():
    url = 'https://www.autoscout24.fr/offres/bmw-320-serie-3-touring-e91-touring-163ch-pack-m-sport-diesel-bleu-671904de-6139-4061-a451-f63bdb61de2b'
    result = await memory_test(url)

if __name__ == "__main__":
    asyncio.run(main())  

httpx内存测试结果 #

测试显示，一个300KB的页面会占用4MB+内存且无法释放，批量处理数千个URL会快速耗尽内存：

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    10  84.7266 MiB  84.7266 MiB           1   @profile(precision=4)
    11                                         async def memory_test(url):  
    12                                               
    13                                          '''
    14                                          async with aiohttp.ClientSession() as session:
    15                                              async with session.get(url) as response:
    16                                                  html = await response.text()    
    17                                                  print(f'Length of response is: {len(html)}')
    18                                          
    19                                          '''
    20  89.1055 MiB   1.8125 MiB           4    async with httpx.AsyncClient(http2=True) as client:
    21                                               
    22  88.2461 MiB   1.7070 MiB          91        html = await client.get(url, follow_redirects=True)
    23  89.1055 MiB   0.8594 MiB           1        print(f'Length of response is: {len(html.text)}')   
    24                                          
    25                                          
    26  89.1055 MiB   0.0000 MiB           1    del html    
    27  89.1055 MiB   0.0000 MiB           1    return None

aiohttp内存测试结果 #

切换至aiohttp则无此问题：

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    10  84.6523 MiB  84.6523 MiB           1   @profile(precision=4)
    11                                         async def memory_test(url):  
    12                                               
    13                                          
    14  88.2812 MiB   0.0000 MiB           3    async with aiohttp.ClientSession() as session:
    15  88.2812 MiB   2.2344 MiB           7        async with session.get(url) as response:
    16  88.2812 MiB   1.3945 MiB           3            html = await response.text()    
    17  88.2812 MiB   0.0000 MiB           1            print(f'Length of response is: {len(html)}')
    18                                          
    19  88.2812 MiB   0.0000 MiB           1    '''
    20                                          async with httpx.AsyncClient(http2=True) as client:
    21                                               
    22                                              html = await client.get(url, follow_redirects=True)
    23                                              print(f'Length of response is: {len(html.text)}')   
    24                                          '''
    25                                          
    26  87.6484 MiB  -0.6328 MiB           1    del html    
    27  87.6484 MiB   0.0000 MiB           1    return None

结论与建议 #

从测试数据对比来看，这更倾向于是httpx的潜在问题，而非对Python内存管理的预期不合理：

• httpx在HTTP2异步请求后，内存占用未随del html操作回落，存在未被正确释放的内存引用；
• aiohttp的内存占用在请求完成后能正常回落，符合Python垃圾回收的预期表现。

此外，httpx官方社区已有其他开发者反馈类似的内存泄漏问题，尤其是在HTTP2模式下的异步请求场景，进一步验证这是库层面的潜在问题。

临时缓解方案可尝试：

• 复用httpx.AsyncClient实例，避免每次请求新建客户端（当前代码每次请求都创建新客户端，会额外增加内存开销）；
• 在批量请求间隙手动调用gc.collect()强制触发垃圾回收；
• 关注httpx官方的修复进展，等待后续版本更新。

内容的提问来源于stack exchange，提问作者splotsh