亚马逊商品页爬虫价格定位NoneType问题 多CSS选择器适配方案咨询

实现思路 #

• 按要求的优先级预先定义所有价格定位规则，依次尝试匹配
• 每次调用find()方法后先判断返回的元素是否为空，非空才执行文本提取操作
• 所有规则均匹配失败时返回默认值，避免空对象调用属性报错

修改后完整代码 #

import requests
from bs4 import BeautifulSoup
import pandas as pd

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36"}
urls = [f'https://www.amazon.co.uk/dp/B083PHB6XX', f'https://www.amazon.co.uk/dp/B089SQHDMR', f'https://www.amazon.co.uk/dp/B0813QVVSW']
# 按优先级定义价格选择器列表，后续新增规则直接往列表加即可
price_selectors = [
    {'tag': 'span', 'attrs': {'id': 'priceblock_ourprice'}},
    {'tag': 'span', 'attrs': {'id': 'priceblock_pospromoprice'}},
    {'tag': 'span', 'attrs': {'class': 'a-size-base a-color-price'}}
]

for url in urls:
    # 可选：加请求异常捕获，避免网络问题导致程序终止
    try:
        r = requests.get(url, headers=headers, timeout=10)
        r.raise_for_status()
    except Exception as e:
        print(f"请求{url}失败：{str(e)}")
        continue
    soup = BeautifulSoup(r.content, 'lxml')
    
    # 商品名也加空判断，避免特殊页面无商品标题报错
    name_elem = soup.find('span', {'id': 'productTitle'})
    name = name_elem.text.strip() if name_elem else '商品标题未找到'
    
    # 多选择器依次匹配价格
    price = '暂无可用价格'
    for selector in price_selectors:
        price_elem = soup.find(selector['tag'], selector['attrs'])
        if price_elem:
            price = price_elem.text.strip()
            break

    print(f"商品名：{name}")
    print(f"价格：{price}\n")

逻辑说明 #

• 优先级控制：price_selectors列表的顺序就是匹配顺序，完全符合优先匹配priceblock_ourprice、其次匹配促销价、最后匹配通用价格class的需求
• 空值兼容：所有元素查找后都加了非空判断，不会出现NoneType调用text属性的报错
• 可扩展性：后续如果发现其他价格定位规则，只需要往price_selectors列表里新增对应配置即可，无需修改判断逻辑

内容的提问来源于stack exchange，提问作者in_d