循环请求URL时第7次必触发ConnectionError问题求助

我有一个包含url列的DataFrame，尝试向这些URL对应的服务器发送请求并提取页面中的<p>元素。运行脚本时总会在第7次请求时触发错误：

ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

即使修改起始索引（比如启用k+=5），之前出错的URL能成功请求，但新的第7次请求仍会触发相同错误，无法确定原因。

相关代码 #

blocklist = [
    'style',
    'script',
    'meta',
    'head'
    # other elements,
]

for k,i in enumerate(df['url']):   
    #k+=5
    website_text=list()
    print(df.at[k,'url'])   
    response=requests.get(i)
    soup = BeautifulSoup(response.content, 'html.parser')
    if soup.findAll('p'):                          
        for data in soup.find_all("p"): 
            #print(data.get_text(),'\n','=================================================================================================','\n')                          
            website_text.append(data.get_text())
        df.at[k,'text']=website_text
                
df.head()    

完整报错信息 #

---------------------------------------------------------------------------
RemoteDisconnected                        Traceback (most recent call last)
File c:\Users\user\anaconda3\envs\GDELT\Lib\site-packages\urllib3\connectionpool.py:790, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    789 # Make the request on the HTTPConnection object
--> 790 response = self._make_request(
    791     conn,
    792     method,
    793     url,
    794     timeout=timeout_obj,
    795     body=body,
    796     headers=headers,
    797     chunked=chunked,
    798     retries=retries,
    799     response_conn=response_conn,
    800     preload_content=preload_content,
    801     decode_content=decode_content,
    802     **response_kw,
    803 )
    805 # Everything went great!

File c:\Users\user\anaconda3\envs\GDELT\Lib\site-packages\urllib3\connectionpool.py:536, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    535 try:
--> 536     response = conn.getresponse()
    537 except (BaseSSLError, OSError) as e:

File c:\Users\user\anaconda3\envs\GDELT\Lib\site-packages\urllib3\connection.py:454, in HTTPConnection.getresponse(self)
    453 # Get the response from http.client.HTTPConnection
--> 454 httplib_response = super().getresponse()
    456 try:

File c:\Users\user\anaconda3\envs\GDELT\Lib\http\client.py:1375, in HTTPConnection.getresponse(self)
   1374 try:
-> 1375     response.begin()
   1376 except ConnectionError:

File c:\Users\user\anaconda3\envs\GDELT\Lib\http\client.py:318, in HTTPResponse.begin(self)
    317 while True:
-> 318     version, status, reason = self._read_status()
    319     if status != CONTINUE:

File c:\Users\user\anaconda3\envs\GDELT\Lib\http\client.py:287, in HTTPResponse._read_status(self)
    284 if not line:
    285     # Presumably, the server closed the connection before
    286     # sending a valid response.
-> 287     raise RemoteDisconnected("Remote end closed connection without"
...
    503 except MaxRetryError as e:
    504     if isinstance(e.reason, ConnectTimeoutError):
    505         # TODO: Remove this in 3.0.0: see #2811

ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

核心原因 #

这个问题大概率是requests依赖的urllib3连接池限制导致的：默认连接池对单域名的并发连接数有约束，连续快速请求时，连接复用机制可能触发服务器的反爬/连接限制；另外未设置浏览器请求头、无请求延时，会让服务器直接判定为自动化请求，触发拦截。

解决步骤 #

1. 添加浏览器请求头：伪装请求来源，降低被拦截概率
2. 设置请求延时：避免短时间内高频请求触发服务器限制
3. 配置连接池与重试机制：提升请求稳定性，自动重试异常请求
4. 添加异常捕获：避免单条请求失败导致整个脚本中断

修改后的代码示例 #

import time
import requests
from bs4 import BeautifulSoup
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

blocklist = [
    'style',
    'script',
    'meta',
    'head'
    # other elements,
]

# 配置会话与重试策略
session = requests.Session()
retry = Retry(
    total=3,  # 总重试次数
    backoff_factor=1,  # 重试间隔递增（1s→2s→4s）
    status_forcelist=[429, 500, 502, 503, 504]  # 需要重试的HTTP状态码
)
adapter = HTTPAdapter(max_retries=retry, pool_connections=20, pool_maxsize=20)
session.mount('http://', adapter)
session.mount('https://', adapter)

# 模拟Chrome浏览器请求头
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}

for k, url in enumerate(df['url']):   
    website_text = []
    print(df.at[k, 'url'])   
    try:
        # 添加2秒延时，避免请求过快
        time.sleep(2)
        response = session.get(url, headers=headers, timeout=10)
        response.raise_for_status()  # 主动触发HTTP错误状态码的异常
        soup = BeautifulSoup(response.content, 'html.parser')
        
        if soup.find_all('p'):                          
            for data in soup.find_all("p"): 
                website_text.append(data.get_text())
            df.at[k, 'text'] = website_text
    except requests.exceptions.RequestException as e:
        print(f"第{k}个URL请求失败: {str(e)}")
        df.at[k, 'text'] = None  # 标记失败请求

df.head()

额外说明 #

• 若所有请求针对同一域名，建议把延时调整到3-5秒，进一步降低触发频率限制的概率
• 若问题仍存在，可能是服务器对单IP请求次数做了严格限制，可考虑使用代理IP池或分批次间隔请求

内容的提问来源于stack exchange，提问作者Mostafa Bouzari