HTTP代理异步测试异常排查与优化方案咨询

问题概述 #

• 测试结果不一致：同步requests测试返回4个HTTP 200，异步httpx返回5个，aiohttp全报ClientProxyConnectionError
• httpx超时异常：设置60秒总超时后，某代理测试耗时13480.64秒（电脑休眠后进程仍未终止）

测试输出 #

requests同步测试结果 #

1) --> 185.XXX.XX.XX:80     --> ProxyError      (4.96s)
 2) --> 38.XX.XXX.XXX:443    --> HTTP (200)      (2.50s)
 3) --> 162.XXX.XX.XXX:80    --> HTTP (200)      (20.92s)
 4) --> 18.XXX.XXX.XXX:8080  --> HTTP (200)      (0.61s)
 5) --> 31.XX.XX.XX:50687    --> ConnectionError (7.88s)
 6) --> 177.XX.XXX.XXX:80    --> ProxyError      (21.07s)
 7) --> 8.XXX.XXX.X:4153     --> HTTP (200)      (4.96s)
 8) --> 146.XX.XXX.XXX:12334 --> ProxyError      (21.05s)
 9) --> 67.XX.XXX.XXX:33081  --> ProxyError      (3.03s)
10) --> 37.XXX.XX.XX:80      --> ReadTimeout     (60.16s)
Testing 10 proxies with "requests" took 147.16 seconds.

httpx异步测试结果 #

4) --> 18.XXX.XXX.XXX:8080  --> HTTP (200)          (16.09s)
 2) --> 38.XX.XXX.XXX:443    --> HTTP (200)          (22.11s)
 7) --> 8.XXX.XXX.X:4153     --> HTTP (200)          (12.96s)
 1) --> 185.XXX.XX.XX:80     --> RemoteProtocolError (24.83s)
 9) --> 67.XX.XXX.XXX:33081  --> ConnectError        (6.02s)
 3) --> 162.XXX.XX.XXX:80    --> HTTP (200)          (22.48s)
 6) --> 177.XXX.XXX.XXX:80    --> HTTP (200)          (26.96s)
 5) --> 31.XX.XX.XX:50687    --> ConnectError        (34.50s)
 8) --> 146.XX.XXX.XXX:12334 --> ConnectError        (27.01s)
10) --> 37.XXX.XX.XX:80      --> ReadError           (13480.64s)
Testing 10 proxies with "httpx" took 13507.80 seconds.

aiohttp异步测试结果 #

1) --> 185.XXX.XX.XX:80     --> ClientProxyConnectionError  (1.30s)
 2) --> 38.XX.XXX.XXX:443    --> ClientProxyConnectionError  (0.67s)
 3) --> 162.XXX.XX.XXX:80    --> ClientProxyConnectionError  (0.77s)
 4) --> 18.XXX.XXX.XXX:8080  --> ClientProxyConnectionError  (0.83s)
 5) --> 31.XX.XX.XX:50687    --> ClientProxyConnectionError  (0.85s)
 6) --> 177.XX.XXX.XXX:80    --> ClientProxyConnectionError  (0.91s)
 7) --> 8.XXX.XXX.X:4153     --> ClientProxyConnectionError  (0.94s)
 8) --> 146.XX.XXX.XXX:12334 --> ClientProxyConnectionError  (1.03s)
 9) --> 67.XX.XXX.XXX:33081  --> ClientProxyConnectionError  (1.05s)
10) --> 37.XXX.XX.XX:80      --> ClientProxyConnectionError  (0.62s)
Testing 10 proxies with "aiohttp" took 2.42 seconds.

代码问题排查 #

1. httpx异步测试核心问题 #

• 变量引用错误：is_alive_httpx函数中打印语句使用了未定义的变量i，实际参数为index，存在运行时错误风险
• 超时配置不严谨：仅设置总超时TIMEOUT，未细分连接、读取等阶段的超时，电脑休眠导致TCP连接假死时，无法及时终止请求
• 资源浪费：每个代理请求新建一个AsyncClient，增加连接开销和资源占用
• 异常捕获范围过宽：捕获所有Exception，可能掩盖代码逻辑错误（如变量未定义）

2. aiohttp异步测试核心问题 #

• Session管理错误：在async with ClientSession上下文内手动调用await client.close()，会导致会话重复关闭，引发潜在错误
• 变量未初始化：color变量仅在try块内定义，异常时未赋值，会触发UnboundLocalError
• 环境干扰：trust_env=True会读取系统代理配置，可能与测试目标代理冲突
• 异常处理逻辑混乱：异常块内打印错误信息，但后续仍执行正常流程的打印语句，导致输出混乱

3. 通用问题 #

• 无并发限制：异步测试时无限制并发请求，可能触发目标网站或代理的限流机制，导致测试结果失真
• 代理未预校验：未过滤无效格式的代理（如空行、非ip:port格式），增加无效请求

优化建议 #

通用优化措施 #

• 限制并发数：使用asyncio.Semaphore控制异步请求并发量（建议10-20），避免触发限流
• 代理预校验：测试前通过正则^\d+\.\d+\.\d+\.\d+:\d+$过滤格式无效的代理
• 统一配置：三个库的请求头、超时、重定向策略保持完全一致，确保测试结果可比性

httpx优化代码示例 #

async def is_alive_httpx(index: int, proxy: str, sem: asyncio.Semaphore, domain: str = DEFAULT_DOMAIN) -> None:
    async with sem:
        color = "\033[91m"
        start = time.perf_counter()
        status = "UnknownError"
        # 细化超时配置，覆盖连接、读取等阶段
        timeout = httpx.Timeout(total=TIMEOUT, connect=10, read=30)
        try:
            async with httpx.AsyncClient(timeout=timeout, headers=HEADERS, follow_redirects=True) as session:
                response = await session.get(f"http://{domain}", proxy=f"http://{proxy}")
                status = f"HTTP ({response.status_code})"
                if response.status_code == 200:
                    color = "\033[92m"
        except httpx.HTTPError as exception:
            status = type(exception).__name__
        finally:
            print(f"{index:>2}) --> {color+proxy:30}\033[0m --> {status:20}\t({time.perf_counter()-start:.2f}s)")

async def test_proxies(proxies_list: list[str], func):
    if proxies_list:
        sem = asyncio.Semaphore(10)
        await asyncio.gather(*[func(ip[0], ip[1], sem) for ip in enumerate(proxies_list, 1)])

aiohttp优化代码示例 #

async def is_alive_aiohttp(index: int, proxy: str, sem: asyncio.Semaphore, domain: str = DEFAULT_DOMAIN) -> None:
    async with sem:
        color = "\033[91m"
        start = time.perf_counter()
        status = "UnknownError"
        try:
            # 禁用系统代理干扰，由async with自动管理会话关闭
            async with aiohttp.ClientSession(
                timeout=aiohttp.ClientTimeout(total=TIMEOUT),
                headers=HEADERS,
                trust_env=False,
                connector=aiohttp.TCPConnector(force_close=True, limit_per_host=5)
            ) as client:
                response = await client.get(f"http://{domain}", proxy=f"http://{proxy}")
                status = f"HTTP ({response.status})"
                if response.status == 200:
                    color = "\033[92m"
        except aiohttp.ClientError as exception:
            status = type(exception).__name__
        finally:
            print(f"{index:>2}) --> {color+proxy:30}\033[0m --> {status:26}\t({time.perf_counter()-start:.2f}s)")
            await asyncio.sleep(0.1)  # 轻微延迟避免请求过载

其他优化点 #

• httpx超时增强：设置keepalive_expiry缩短连接存活时间，避免假死连接占用资源
• 结果持久化：将有效代理写入文件，避免重复测试
• 日志替代打印：使用logging模块替代print，便于后续排查问题

内容的提问来源于stack exchange，提问作者Mycroft_47