asyncio.gather()触发TypeError，代理检查器无法返回有效代理列表

问题概述 #

基于AsyncHTMLSession实现的代理检查器，目标返回有效代理列表，但运行时报错且事件循环无法正常终止。

原代码 #

import asyncio
from bs4 import BeautifulSoup
from requests_html import AsyncHTMLSession

async def fetch_proxies(url):
    session = AsyncHTMLSession()
    webpage = await session.get(url) # headers=headers
    await session.close()
    soup = BeautifulSoup(webpage.html.raw_html, 'lxml')
    tag = soup.find('textarea', {'class':'form-control'})
    return tag.text

async def check_valid_proxy(proxy):
    try:
        session = AsyncHTMLSession()
        webpage = await session.get('https://ifconfig.me', proxies = {'https' : proxy})
        await session.close()
        return proxy
    except Exception as err:
        print(err)

async def main():
    url = 'https://free-proxy-list.net'
    proxy_fetch = await fetch_proxies(url)
    proxy_list = proxy_fetch.split('\n')[3:-1]
    tasks = []
    for proxy in proxy_list:
        task = asyncio.create_task(check_valid_proxy(proxy))
        tasks.append(task)
        tasks.append(await asyncio.sleep(.1))
    valid_proxies = asyncio.gather(*tasks)
    return valid_proxies

loop = asyncio.get_event_loop()
valid_proxy_list = loop.run_until_complete(main())
print(valid_proxy_list)

报错信息 #

TypeError错误

Traceback (most recent call last):
  File "/home/mu0/Documents/python_files/web_automation/mailSpider/proxy.py", line 36, in <module>
    valid_proxy_list = loop.run_until_complete(main())
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/mu0/Documents/python_files/web_automation/mailSpider/proxy.py", line 32, in main
    valid_proxies = asyncio.gather(*tasks)
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/tasks.py", line 817, in gather
    fut = _ensure_future(arg, loop=loop)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/tasks.py", line 664, in _ensure_future
    raise TypeError('An asyncio.Future, a coroutine or an awaitable '
TypeError: An asyncio.Future, a coroutine or an awaitable is required

事件循环无法终止错误

Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1553, in _shutdown
    atexit_call()
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 31, in _python_exit
    t.join()
  File "/usr/lib/python3.11/threading.py", line 1112, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.11/threading.py", line 1132, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt: 

错误原因分析 #

1. tasks列表混入非可等待对象：tasks.append(await asyncio.sleep(.1))中，await asyncio.sleep(.1)直接执行后返回None，将None加入tasks列表后，asyncio.gather(*tasks)无法处理非可等待对象，触发TypeError。
2. 未等待gather结果：asyncio.gather(*tasks)返回的是Future对象，main函数直接返回该对象，导致事件循环未等待所有检查任务完成就结束，引发线程关闭阻塞。
3. 代理格式不规范：从free-proxy-list.net获取的代理为IP:PORT格式，requests_html的proxies参数要求代理URL包含协议前缀（如http://IP:PORT），否则会导致连接失败。
4. 事件循环使用方式过时：Python3.7+推荐使用asyncio.run()替代asyncio.get_event_loop().run_until_complete()，后者易出现资源未正确释放的问题。

修复后的代码 #

import asyncio
from bs4 import BeautifulSoup
from requests_html import AsyncHTMLSession

async def fetch_proxies(url):
    async with AsyncHTMLSession() as session:
        webpage = await session.get(url)
        soup = BeautifulSoup(webpage.html.raw_html, 'lxml')
        tag = soup.find('textarea', {'class':'form-control'})
        return tag.text

async def check_valid_proxy(proxy):
    # 为代理添加协议前缀，符合requests_html参数要求
    proxy_url = f'http://{proxy}'
    try:
        async with AsyncHTMLSession() as session:
            # 设置超时，避免单个检查长时间阻塞
            webpage = await session.get('https://ifconfig.me', proxies={'https': proxy_url}, timeout=10)
            return proxy
    except Exception:
        # 可注释该行以减少控制台输出
        # print(f"Proxy {proxy} invalid")
        return None

async def main():
    url = 'https://free-proxy-list.net'
    proxy_fetch = await fetch_proxies(url)
    # 拆分列表并过滤空字符串，避免无效代理检查
    proxy_list = [p.strip() for p in proxy_fetch.split('\n') if p.strip()][2:-1]
    
    tasks = []
    for proxy in proxy_list:
        task = asyncio.create_task(check_valid_proxy(proxy))
        tasks.append(task)
        # 控制任务提交间隔，避免触发目标网站反爬
        await asyncio.sleep(0.1)
    
    # 等待所有任务完成，过滤无效代理
    results = await asyncio.gather(*tasks)
    valid_proxies = [proxy for proxy in results if proxy is not None]
    return valid_proxies

# 使用Python3.7+推荐的方式启动事件循环
if __name__ == "__main__":
    valid_proxy_list = asyncio.run(main())
    print("Valid proxies:")
    for proxy in valid_proxy_list:
        print(proxy)

关键修复点说明 #

1. 移除tasks列表中的无效元素：将tasks.append(await asyncio.sleep(.1))改为直接在循环中await asyncio.sleep(0.1)，实现任务间隔的同时避免向tasks添加非可等待对象。
2. 等待gather结果：使用await asyncio.gather(*tasks)获取所有任务返回值，过滤掉返回None的无效代理，确保返回有效列表。
3. 规范代理格式：为代理添加http://前缀，符合requests_html的参数要求。
4. 使用async with管理session：自动关闭session，避免资源泄漏，解决事件循环无法正常终止的问题。
5. 添加超时设置：在session.get()中设置timeout=10，避免单个代理检查长时间阻塞。
6. 过滤空代理：拆分代理列表时过滤空字符串，减少无效检查。
7. 使用asyncio.run()：简化事件循环管理，确保资源正确释放。

内容的提问来源于stack exchange，提问作者Aleph1