使用Python及Pandas爬取多页网站时遇Remote Connection closed error求解决

这个错误大多是请求频率过高被服务器限制，或是缺少必要请求标识导致服务器拒绝连接，以下是针对性解决办法：

1. 添加请求头模拟浏览器 #

服务器常通过User-Agent识别请求来源，加上浏览器标识能降低被拦截概率。手动构造请求获取页面内容后，再传给pd.read_html：

import pandas as pd
import requests
from time import sleep

url_link = 'https://www.taneps.go.tz/epps/viewAllAwardedContracts.do?d-3998960-p={}&selectedItem=viewAllAwardedContracts.do'
LIST = []
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
}

for number in range(1, 5379):
    url = url_link.format(number)
    try:
        response = requests.get(url, headers=headers, timeout=10)
        response.raise_for_status()  # 检查请求是否成功
        dframe = pd.read_html(response.text, header=None)[0]
        LIST.append(dframe)
        sleep(1)  # 每次请求后延迟，避免高频访问
    except Exception as e:
        print(f"第{number}页爬取失败: {str(e)}")
        sleep(3)  # 失败后延长重试间隔

Result_df = pd.concat(LIST)
Result_df.to_csv('Taneps_contracts.csv')

2. 实现自动重试机制 #

用requests.adapters.HTTPAdapter设置重试规则，自动处理连接中断的情况：

import pandas as pd
import requests
from time import sleep
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

url_link = 'https://www.taneps.go.tz/epps/viewAllAwardedContracts.do?d-3998960-p={}&selectedItem=viewAllAwardedContracts.do'
LIST = []
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
}

# 创建带重试的session
session = requests.Session()
retry = Retry(
    total=3,  # 总重试次数
    backoff_factor=1,  # 重试间隔按1、2、4秒递增
    status_forcelist=[429, 500, 502, 503, 504]  # 需要重试的服务器错误状态码
)
adapter = HTTPAdapter(max_retries=retry)
session.mount('https://', adapter)
session.mount('http://', adapter)

for number in range(1, 5379):
    url = url_link.format(number)
    try:
        response = session.get(url, headers=headers, timeout=10)
        response.raise_for_status()
        dframe = pd.read_html(response.text, header=None)[0]
        LIST.append(dframe)
        sleep(1)
    except Exception as e:
        print(f"第{number}页最终失败: {str(e)}")

Result_df = pd.concat(LIST)
Result_df.to_csv('Taneps_contracts.csv')

3. 优化请求频率 #

如果服务器限制严格，可改用随机延迟避免固定间隔被识别：

from random import uniform
# 替换原sleep语句
sleep(uniform(1, 3))  # 随机延迟1-3秒

额外注意事项 #

• 不要一次性爬取全部5000+页面，可分批次进行，避免长时间占用服务器资源
• 若仍出现错误，检查是否IP被封禁，必要时使用代理IP
• 爬取前查看网站robots.txt，确认允许爬取的范围

内容的提问来源于stack exchange，提问作者tony michael