Python Web爬虫运行报HTTP Error 406: Not Acceptable错误如何解决

问题基本信息 #

• 场景：练习Python网页爬虫、复现教材示例代码时触发报错
• 运行环境：Windows 10操作系统，Anaconda + VS Code开发环境
• 已尝试无效操作：重装urllib库（确认库为预装状态）、添加urllib.error.HTTPError异常捕获逻辑

原始问题代码 #

from urllib import request
from bs4 import BeautifulSoup

page_url = 'https://alansimpson.me/python/scrape_sample.html'
rawpage = request.urlopen(page_url)
soup = BeautifulSoup(rawpage, 'html5lib')
content = soup.article

links_list = []
for link in content.find_all('a'):
    try:
        url = link.get('href')
        img = link.img.get('src')
        text = link.span.text
        links_list.append({'url' : url, 'img' : img, 'text' : text})
    except AttributeError:
        pass

完整报错信息 #

Traceback (most recent call last):
  File "c:\Users\srika\OneDrive\AIO_Python\scraper.py", line 6, in <module>
    rawpage = request.urlopen(page_url)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 523, in open
    response = meth(req, response)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 632, in http_response
    response = self.parent.error(
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 561, in error
    return self._call_chain(*args)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 406: Not Acceptable

报错根因 #

HTTP 406 Not Acceptable错误和代码逻辑、urllib库安装状态无关。urllib默认发起请求时携带的User-Agent字段会标识自身为程序爬虫，目标站点识别到该类非正常浏览器请求后，会直接拒绝返回响应内容。

修复方案 #

给请求添加普通浏览器的User-Agent请求头，模拟真实浏览器访问即可正常获取页面内容，修复后代码如下：

from urllib import request
from bs4 import BeautifulSoup

page_url = 'https://alansimpson.me/python/scrape_sample.html'
# 构造请求头，使用Chrome浏览器的标准User-Agent标识
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
}
# 生成携带自定义请求头的请求对象
req = request.Request(url=page_url, headers=headers)
rawpage = request.urlopen(req)
soup = BeautifulSoup(rawpage, 'html5lib')
content = soup.article

links_list = []
for link in content.find_all('a'):
    try:
        url = link.get('href')
        img = link.img.get('src')
        text = link.span.text
        links_list.append({'url' : url, 'img' : img, 'text' : text})
    except AttributeError:
        pass

# 打印爬取结果验证逻辑正常
print(links_list)

补充说明 #

• urllib是Python官方标准库，随Python环境默认安装，不需要额外执行重装操作，Anaconda环境下不会出现该库缺失的问题
• 仅添加HTTPError异常捕获只能避免程序直接崩溃，无法获取目标页面的正常返回内容，不能从根源解决406类访问被拦截的问题
• 后续爬取其他站点遇到403、406类访问拒绝错误时，优先检查请求头是否缺少正常浏览器的标识字段

内容的提问来源于stack exchange，提问作者Raj K