如何修正Python爬虫代码，仅爬取Waitrose有库存商品的Href？

原代码失效的核心原因有两个：

1. 页面的库存状态可能是JavaScript动态渲染的，requests.get()只能获取初始静态HTML，无法拿到动态加载的缺货标记
2. 缺货元素的定位逻辑不准确，仅通过product.parent.find_all('span')遍历范围太宽泛，容易遗漏或误判

方案1：优化静态元素定位（若库存状态在静态HTML中存在） #

先检查页面静态源码里缺货商品的标记，通常这类网站会给缺货元素加特定类名（比如out-of-stock、stock-status--unavailable）。调整定位逻辑如下：

import requests
import random
from bs4 import BeautifulSoup

user_agent_list = [
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
]
headers = {'User-Agent': random.choice(user_agent_list)}

url = 'https://www.waitrosecellar.com/whisky-shop/view-all-whiskies/whisky-by-brand/macallan'

response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

in_stock_links = []

# 定位每个商品的容器（需根据页面实际class调整）
for product_container in soup.find_all('div', class_="productItem"):
    # 精准匹配缺货标记文本
    out_of_stock_elem = product_container.find('span', string=lambda text: text and "Out of stock" in text.strip())
    # 也可通过class定位：product_container.find('span', class_="outOfStock")
    if not out_of_stock_elem:
        name_elem = product_container.find('div', class_="productName")
        if name_elem and name_elem.a:
            in_stock_links.append(name_elem.a['href'])

print(in_stock_links)

方案2：使用JS渲染获取动态内容（推荐，多数电商站点库存为动态加载） #

原代码导入了requests_html但未使用，利用它可以渲染JS，获取完整页面内容：

import random
from requests_html import HTMLSession

user_agent_list = [
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
]

session = HTMLSession()
headers = {'User-Agent': random.choice(user_agent_list)}

url = 'https://www.waitrosecellar.com/whisky-shop/view-all-whiskies/whisky-by-brand/macallan'

# 渲染JS并等待页面加载完成
response = session.get(url, headers=headers)
response.html.render(sleep=2)

in_stock_links = []

# 遍历所有商品容器
for product_container in response.html.find('div.productItem'):
    # 检查是否包含缺货文本
    out_of_stock = product_container.find('span', containing="Out of stock")
    if not out_of_stock:
        # 获取商品链接
        name_elem = product_container.find('div.productName a', first=True)
        if name_elem:
            in_stock_links.append(name_elem.attrs['href'])

print(in_stock_links)
session.close()

关键优化点 #

• 缩小元素查找范围：从商品容器层级开始遍历，避免无关元素干扰
• 精准匹配缺货标记：用string=lambda或containing直接匹配缺货文本，或通过特定class定位
• 处理动态内容：使用requests_html.render()渲染JS，确保获取完整的页面状态

内容的提问来源于stack exchange，提问作者Ianucci