在Colab环境爬取英国议会网站遭遇403错误的解决咨询

问题描述 #

尝试在Colab环境中访问英国议会下议院议员页面（https://members.parliament.uk/members/commons）时持续出现403错误，网站请求头限制严格，即使配置了完整请求头仍无法解决。

已尝试的代码 #

简单请求头版本：

from urllib.request import urlopen, Request

url = "https://members.parliament.uk/members/commons"

headers={'User-Agent': 'Mozilla/5.0'}

request= Request(url=url, headers=headers) 
response = urlopen(request)
data = response.read()

完整请求头配置版本：

headers = {
  'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15',
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
  'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
  'Accept-Encoding': 'none',
  'Accept-Language': 'en-US,en;q=0.8',
  'Connection': 'keep-alive'
}

可行解决办法 #

1. 用Requests会话补充真实请求头字段 #

网站可能会验证请求的连贯性和更多浏览器特征，使用requests.Session()维持会话，并补充Referer、DNT等字段，更贴近真实浏览器行为：

import requests

url = "https://members.parliament.uk/members/commons"
session = requests.Session()

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Referer': 'https://members.parliament.uk/',
    'DNT': '1',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1'
}

# 可选：手动添加浏览器访问该页面后获取的有效Cookie
# session.cookies.set('SESSION_ID', 'your_cookie_value')

try:
    response = session.get(url, headers=headers)
    response.raise_for_status()
    print("请求成功，页面片段：", response.text[:500])
except requests.exceptions.HTTPError as e:
    print(f"请求失败：{e}")

2. 用Selenium模拟真实浏览器渲染 #

如果网站有JS反爬或行为检测，直接用Selenium启动无头Chrome模拟用户操作，绕过请求头限制（Colab需先安装依赖）：

# 安装依赖
!pip install selenium
!apt-get update
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# 配置无头Chrome
options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36')

# 启动浏览器并访问页面
driver = webdriver.Chrome(options=options)
driver.get("https://members.parliament.uk/members/commons")

print("页面片段：", driver.page_source[:500])
driver.quit()

3. 更换代理IP（Colab环境适配） #

Colab的公用IP可能被网站列入黑名单，使用可靠的代理IP更换请求来源，示例（需替换为有效代理）：

import requests

url = "https://members.parliament.uk/members/commons"
proxies = {
    'http': 'http://your-proxy-ip:port',
    'https': 'https://your-proxy-ip:port'
}

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}

try:
    response = requests.get(url, headers=headers, proxies=proxies)
    response.raise_for_status()
    print(response.text[:500])
except requests.exceptions.HTTPError as e:
    print(f"请求失败：{e}")

内容的提问来源于stack exchange，提问作者amatsuo_net