网页爬虫每次运行输出不一致并报错，请求协助排查

问题描述 #

以下是基于urllib和BeautifulSoup的Newegg GTX产品列表爬虫代码：

from urllib import request
from bs4 import BeautifulSoup
page_url = "http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=-1&IsNodeId=1&Description=GTX&bop=And&Page=1&PageSize=36&order=BESTMATCH"
uclient = request.urlopen(page_url) #open a webclient
html_page = uclient.read()
page_soup = BeautifulSoup(html_page,"html.parser") 
uclient.close()
containers = page_soup.find_all("div",{"class":"item-cell"}) 
title_list = []
for contain in containers:
    title = contain.select("img")[0]["title"]
    print(title)# for troubleshooting
    print(len(title_list)) #for troubleshooting
    title_list.append(title)
print(title_list)

每次运行该代码时，返回的标题数量不稳定（有时12条，有时28条、30条），随后会抛出如下错误：

Input In [67], in <cell line: 16>()
     15 title_list = []
     16 for contain in containers:
---&gt; 17     title = contain.select("img")[0]["title"]
     18     print(title)# for troubleshooting
     19     print(len(title_list)) #for troubleshooting

File C:\ProgramData\Anaconda3\lib\site-packages\bs4\element.py:1519, in Tag.__getitem__(self, key)
   1516 def __getitem__(self, key):
   1517     """tag[key] returns the value of the 'key' attribute for the Tag,
   1518     and throws an exception if it's not there."""
-&gt; 1519     return self.attrs[key]

KeyError: 'title'

问题原因 #

• 反爬拦截：Newegg服务器会识别无请求头的爬虫请求，返回不完整的页面内容，导致每次抓取到的item-cell数量不一致，部分元素甚至缺失title属性。
• 代码健壮性不足：直接通过contain.select("img")[0]["title"]访问元素，未做异常判断——一旦某个item-cell中没有img标签，或img标签没有title属性，就会触发错误。

修复方案 #

1. 添加请求头模拟浏览器 #

给请求添加User-Agent等头信息，避免被反爬识别：

from urllib import request
from bs4 import BeautifulSoup

page_url = "http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=-1&IsNodeId=1&Description=GTX&bop=And&Page=1&PageSize=36&order=BESTMATCH"
# 模拟浏览器请求头
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}
# 构造带请求头的请求
req = request.Request(page_url, headers=headers)
uclient = request.urlopen(req) 
html_page = uclient.read()
page_soup = BeautifulSoup(html_page,"html.parser") 
uclient.close()

2. 增加异常判断提升代码健壮性 #

遍历容器时，先检查img标签是否存在，再验证title属性：

containers = page_soup.find_all("div",{"class":"item-cell"}) 
title_list = []
for contain in containers:
    img_tags = contain.select("img")
    # 先判断是否找到img标签
    if img_tags:
        target_img = img_tags[0]
        # 检查title属性是否存在
        if 'title' in target_img.attrs:
            title = target_img['title']
            title_list.append(title)
            print(title)
            print(len(title_list))
print(title_list)

3. 可选：使用更稳定的HTML解析器 #

Python内置的html.parser稳定性不如第三方解析器，可安装lxml或html5lib替换：

# 先安装依赖：pip install lxml
page_soup = BeautifulSoup(html_page, "lxml")

内容的提问来源于stack exchange，提问作者Asi