运行aiogram Telegram机器人时遭遇HTTP连接池超时问题求助

问题背景 #

代码在本地运行正常，但部署到cPanel和Heroku时出现HTTP连接超时错误，错误信息如下：

HTTPConnectionPool(host='example.com', port=80): Max retries exceeded with url: /bla/bla (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x2b2341447280>, 'Connection to example.com timed out. (connect timeout=3)'))

相关核心代码：

import requests
from bs4 import BeautifulSoup
from aiogram import types
from data.config import ADMINS
from loader import dp, db, bot

link = 'http://example.com/'

@dp.message_handler(text="✅ Confirm (1-step)", user_id=ADMINS, state="*")
async def scrapping1(message: types.Message):
    await message.answer("Confirmed ✅")
    with requests.Session() as s:
        s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36'
        res = s.get(link, timeout=3)
        soup = BeautifulSoup(res.text,'html.parser')   
        payload = {i['name']:i.get('value','') for i in soup.select('input[name]')}         
        payload['LoginForm[username]'] = "blabla"
        payload['LoginForm[password]'] = "blabla"

        print(payload)
        s.post(link,data=payload, timeout=3)
        for id in range(10, 100):
            try:
                r = s.get(f'http://example.com?id={id}', timeout=1)
                response = BeautifulSoup(r.text, "html.parser") 
                # 省略后续处理代码
            except Exception as e:
                print(f"请求ID {id} 失败: {e}")

解决方案 #

1. 增加超时时间并添加自动重试机制 #

利用urllib3的重试工具配合requests的适配器，实现请求失败后的自动重试，避免单次超时直接报错：

import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
# 其他导入保持不变

@dp.message_handler(text="✅ Confirm (1-step)", user_id=ADMINS, state="*")
async def scrapping1(message: types.Message):
    await message.answer("Confirmed ✅")
    with requests.Session() as s:
        # 配置重试策略
        retry_strategy = Retry(
            total=3,  # 总重试次数
            backoff_factor=1,  # 重试间隔递增（1s→2s→4s）
            status_forcelist=[429, 500, 502, 503, 504],  # 需要重试的状态码
            allowed_methods=["GET", "POST"]  # 允许重试的请求方法
        )
        adapter = HTTPAdapter(max_retries=retry_strategy)
        s.mount("http://", adapter)
        s.mount("https://", adapter)

        s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36'
        
        # 调整超时时间（根据实际网络情况设置，比如10s）
        res = s.get(link, timeout=10)
        soup = BeautifulSoup(res.text,'html.parser')   
        payload = {i['name']:i.get('value','') for i in soup.select('input[name]')}         
        payload['LoginForm[username]'] = "blabla"
        payload['LoginForm[password]'] = "blabla"

        print(payload)
        s.post(link, data=payload, timeout=10)
        for id in range(10, 100):
            try:
                r = s.get(f'http://example.com?id={id}', timeout=5)
                response = BeautifulSoup(r.text, "html.parser") 
                # 后续处理代码
            except requests.exceptions.RequestException as e:
                print(f"处理ID {id} 时出错: {str(e)}")
                # 可选：给管理员发送报错提醒
                await bot.send_message(ADMINS[0], f"ID {id} 请求失败: {str(e)}")

2. 检查环境网络限制 #

• cPanel和Heroku可能对出站HTTP请求有IP限制，确认目标网站example.com是否允许这些平台的IP段访问，若需要可配置代理：

# 在Session初始化后添加代理配置
proxies = {
    'http': 'http://your-proxy-address:port',
    'https': 'http://your-proxy-address:port'
}
s.proxies.update(proxies)

3. 优化请求频率 #

循环中高频请求可能触发目标服务器限制，添加请求间隔：

import asyncio
# 在循环内部添加延迟
for id in range(10, 100):
    try:
        r = s.get(f'http://example.com?id={id}', timeout=5)
        response = BeautifulSoup(r.text, "html.parser") 
        # 后续处理代码
        await asyncio.sleep(0.5)  # 每次请求后延迟0.5秒
    except requests.exceptions.RequestException as e:
        print(f"处理ID {id} 时出错: {str(e)}")

内容的提问来源于stack exchange，提问作者Jamshid