使用requests批量请求页面时无响应/无状态码及503问题的排查与解决

成因 #

• 反爬机制拦截：网站能识别出requests库默认的请求标识（如python-requests/x.x.x的User-Agent），判定为非人类访问，会对后续请求做限制——要么返回503服务不可用，要么直接挂起连接不响应。
• 请求频率过高：短时间内连续发送无间隔的请求，触发网站的限流规则，后续请求被拦截。
• 缺少必要请求头：浏览器访问时会携带Cookie、Referer、User-Agent等头信息，你的请求没有这些，不符合网站的正常访问特征。

解决方法 #

1. 模拟浏览器请求头 #

给请求添加符合浏览器特征的User-Agent（可从自己浏览器的开发者工具中复制），必要时补充Cookie等信息：

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
}

urls = [
    "https://www.baha.com/nasdaq-100-index/index/tts-751307/name/asc/1/index/performance/471",
    "https://www.baha.com/adobe/stocks/details/tts-117450574",
    "https://www.baha.com/advanced-micro-devices/stocks/details/tts-117449963",
    "https://www.baha.com/airbnb-inc/stocks/details/tts-208432020",
    "https://www.baha.com/alphabet-a/stocks/details/tts-117453820",
    "https://www.baha.com/alphabet-c/stocks/details/tts-117453810"
]

for url in urls:
    web = requests.get(url, headers=headers)
    print(web.status_code)

2. 添加请求间隔 #

在每次请求后暂停一段时间，避免请求过于密集：

import requests
import time

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
}

urls = [
    "https://www.baha.com/nasdaq-100-index/index/tts-751307/name/asc/1/index/performance/471",
    "https://www.baha.com/adobe/stocks/details/tts-117450574",
    "https://www.baha.com/advanced-micro-devices/stocks/details/tts-117449963",
    "https://www.baha.com/airbnb-inc/stocks/details/tts-208432020",
    "https://www.baha.com/alphabet-a/stocks/details/tts-117453820",
    "https://www.baha.com/alphabet-c/stocks/details/tts-117453810"
]

for url in urls:
    web = requests.get(url, headers=headers)
    print(web.status_code)
    time.sleep(2)  # 暂停2秒，可根据网站限制调整时长

3. 使用会话保持 #

用requests.Session()维持会话状态，模拟浏览器的连续访问逻辑：

import requests
import time

session = requests.Session()
session.headers.update({
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
})

urls = [
    "https://www.baha.com/nasdaq-100-index/index/tts-751307/name/asc/1/index/performance/471",
    "https://www.baha.com/adobe/stocks/details/tts-117450574",
    "https://www.baha.com/advanced-micro-devices/stocks/details/tts-117449963",
    "https://www.baha.com/airbnb-inc/stocks/details/tts-208432020",
    "https://www.baha.com/alphabet-a/stocks/details/tts-117453820",
    "https://www.baha.com/alphabet-c/stocks/details/tts-117453810"
]

for url in urls:
    web = session.get(url)
    print(web.status_code)
    time.sleep(2)

4. 增加重试机制 #

针对503这类临时错误，添加自动重试逻辑：

import requests
import time
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
# 配置重试策略
retry_strategy = Retry(
    total=3,
    status_forcelist=[503],
    backoff_factor=1  # 重试间隔依次为1s、2s、4s
)
adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount("https://", adapter)
session.headers.update({
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
})

urls = [
    "https://www.baha.com/nasdaq-100-index/index/tts-751307/name/asc/1/index/performance/471",
    "https://www.baha.com/adobe/stocks/details/tts-117450574",
    "https://www.baha.com/advanced-micro-devices/stocks/details/tts-117449963",
    "https://www.baha.com/airbnb-inc/stocks/details/tts-208432020",
    "https://www.baha.com/alphabet-a/stocks/details/tts-117453820",
    "https://www.baha.com/alphabet-c/stocks/details/tts-117453810"
]

for url in urls:
    try:
        web = session.get(url, timeout=10)
        print(web.status_code)
        time.sleep(2)
    except requests.exceptions.RequestException as e:
        print(f"请求{url}失败: {e}")

总结 #

核心是模拟正常浏览器的访问行为：匹配请求头、控制请求频率、维持会话状态，同时通过重试机制处理临时错误，避免被网站的反爬和限流规则拦截。

内容的提问来源于stack exchange，提问作者M14