Python爬取Wild Fork Foods牛肉价格返回None问题求助

问题背景 #

需要爬取URL https://wildforkfoods.com/products/usda-choice-black-angus-beef-thick-ribeye-steak?variant=8056461918244 的牛肉价格，期望输出 [{"price": "$18.73"}]，但实际输出始终为 [{"price": None}]，原代码如下：

import requests
from bs4 import BeautifulSoup

url = 'https://wildforkfoods.com/products/usda-choice-black-angus-beef-thick-ribeye-steak?variant=8056461918244'

headers = {
  'authority': 'p.yotpo.com',
  'accept': 'image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8',
  'accept-language': 'en,zh-CN;q=0.9,zh;q=0.8,ja-CN;q=0.7,ja;q=0.6',
  'cookie': 'pixel=051f2bf3-4b41-4f05-4f39-58fe53c3574f; pixel=051f2bf3-4b41-4f05-4f39-58fe53c3574f',
  'referer': 'https://wildforkfoods.com/',
  'sec-ch-ua': '"Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"',
  'sec-ch-ua-mobile': '?0',
  'sec-ch-ua-platform': '"macOS"',
  'sec-fetch-dest': 'image',
  'sec-fetch-mode': 'no-cors',
  'sec-fetch-site': 'cross-site',
  'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36'
}
resp = requests.get(url, headers=headers)

soup = BeautifulSoup(resp.text,'html.parser')
l=[]
obj={}
try:
    obj["price"] = soup.find("span",{"itemprop":"price"}).text.replace("Now ","")
except:
    obj["price"]=None
try:
    obj["name"] = soup.find("h1",{"itemprop":"name"}).text
except:
    obj["name"]=None
try:
    obj["rating"] = soup.find("span",{"class":"rating-number"}).text.replace("(","").replace(")",""")
except:
    obj["rating"]=None
    
    l.append(obj)
print(l)

问题排查 #

1. 请求头配置错误：原headers中的authority指向第三方评论服务域名p.yotpo.com，而非目标网站wildforkfoods.com，同时sec-fetch-dest设为image，服务器判定为图片请求，返回的页面内容不完整，导致无法找到价格元素。
2. 元素定位失效：目标网站页面结构已变化，原代码依赖的span[itemprop="price"]元素不存在，定位方式失效。
3. 语法与缩进错误：rating字段的replace方法存在语法错误（多了一个双引号）；l.append(obj)缩进在rating的except块内，只有当评分获取失败时才会将对象加入列表，逻辑错误。

修复方案 #

修复后的代码 #

import requests
from bs4 import BeautifulSoup
import json

url = 'https://wildforkfoods.com/products/usda-choice-black-angus-beef-thick-ribeye-steak?variant=8056461918244'

# 适配目标网站的请求头
headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36',
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'accept-language': 'en,zh-CN;q=0.9,zh;q=0.8',
    'referer': 'https://wildforkfoods.com/',
    'sec-fetch-dest': 'document',
    'sec-fetch-mode': 'navigate',
    'sec-fetch-site': 'same-origin',
    'authority': 'wildforkfoods.com'
}

resp = requests.get(url, headers=headers)
resp.encoding = 'utf-8'  # 确保页面编码正确

soup = BeautifulSoup(resp.text, 'html.parser')
result_list = []
product_obj = {}

# 优先从页面结构化数据（JSON-LD）提取价格，稳定性更高
try:
    script_tag = soup.find('script', type='application/ld+json')
    if script_tag:
        product_data = json.loads(script_tag.string)
        product_obj["price"] = f"${product_data['offers']['price']}"
except Exception:
    product_obj["price"] = None

# 结构化数据获取失败时，通过HTML元素类名备选提取
if not product_obj["price"]:
    try:
        # 先尝试获取促销价
        price_span = soup.find('span', class_='price-item price-item--sale')
        if price_span:
            product_obj["price"] = price_span.text.strip()
        else:
            # 再尝试获取常规价
            price_span = soup.find('span', class_='price-item price-item--regular')
            product_obj["price"] = price_span.text.strip() if price_span else None
    except Exception:
        product_obj["price"] = None

# 修复名称与评分的提取逻辑及语法错误
try:
    product_obj["name"] = soup.find("h1", {"itemprop": "name"}).text.strip()
except:
    product_obj["name"] = None

try:
    rating_span = soup.find("span", {"class": "rating-number"})
    product_obj["rating"] = rating_span.text.replace("(", "").replace(")", "") if rating_span else None
except:
    product_obj["rating"] = None

# 修正缩进，确保对象始终被加入结果列表
result_list.append(product_obj)
print(result_list)

关键修复点 #

• 修正请求头：将authority改为目标网站域名，调整sec-fetch-dest为document，确保服务器返回完整的商品页面。
• 稳定提取价格：优先从页面嵌入的JSON-LD结构化数据中提取价格，这种方式不受页面样式修改影响；同时保留HTML元素定位作为备选方案。
• 修复语法与缩进：修正replace方法的语法错误，将append操作移到所有逻辑之外，确保结果列表始终包含商品对象。

内容的提问来源于stack exchange，提问作者yutong jin