Python Requests库请求网站无响应且持续运行的问题求助

1. 给请求添加超时限制，避免无限等待 #

程序一直无响应大概率是连接超时，先加超时参数定位问题：

import requests

req = requests.session()
url = "https://www.stepstone.de/"
try:
    # 设置10秒超时，包含连接和读取两个阶段
    response = req.get(url, timeout=10)
    print(response.status_code)
except requests.exceptions.Timeout:
    print("请求超时，检查网络或者更换IP尝试")
except Exception as e:
    print(f"请求出错: {str(e)}")

2. 补全完整请求头信息 #

网站会校验请求头完整性，仅加User-Agent可能不够，补上常用字段：

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.5",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1"
}

req = requests.session()
url = "https://www.stepstone.de/"
try:
    response = req.get(url, headers=headers, timeout=10)
    print(response.status_code)
except Exception as e:
    print(f"请求出错: {str(e)}")

3. 使用代理IP访问 #

如果你的IP被网站拦截，更换代理IP尝试（需自行准备可靠代理服务）：

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
}

proxies = {
    "http": "http://你的代理地址:端口",
    "https": "https://你的代理地址:端口"
}

req = requests.session()
url = "https://www.stepstone.de/"
try:
    response = req.get(url, headers=headers, proxies=proxies, timeout=10)
    print(response.status_code)
except Exception as e:
    print(f"请求出错: {str(e)}")

4. 临时关闭SSL验证排查问题 #

部分网站的SSL证书可能导致连接失败，可临时关闭验证测试（生产环境不建议，存在安全风险）：

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
}

req = requests.session()
url = "https://www.stepstone.de/"
try:
    response = req.get(url, headers=headers, timeout=10, verify=False)
    print(response.status_code)
except Exception as e:
    print(f"请求出错: {str(e)}")

5. 用模拟浏览器绕过反爬验证 #

如果以上方法都无效，说明网站启用了Cloudflare这类人机验证，requests无法直接绕过，可使用selenium模拟浏览器访问：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time

options = Options()
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")

# 需提前安装对应浏览器驱动（如ChromeDriver）并配置路径
driver = webdriver.Chrome(options=options)
driver.get("https://www.stepstone.de/")
time.sleep(3)  # 等待页面加载完成
print(driver.title)
driver.quit()

内容的提问来源于stack exchange，提问作者Soroush