如何用Python的BeautifulSoup4爬取eBay指定字段？

问题说明 #

跟着过时视频学习网页爬取，适配失效代码时已调整部分后端变更，但无法提取售出日期字段（已知位于<div class="s-item__title--tag">标签内），需适配代码提取以下目标字段：

• 标题
• 售出日期
• 售出价格
• 购买地区（非发货地）
  附加需求：
• 商品listing链接

当前脚本为ebay_scraper.py，使用方式：

python ebay_scraper.py [以加号分隔空格的搜索词]

示例：

python ebay_scraper.py darth+vader > output.tsv

适配后的完整代码 #

import sys, requests
from bs4 import BeautifulSoup
import pandas as pd

def main():
    searchterm = sys.argv[1]
    # 修正URL参数，移除多余的+1，确保搜索词正确传递
    url = f"https://www.ebay.co.uk/sch/i.html?_from=R40&_nkw={searchterm}&_sacat=0&LH_TitleDesc=0&LH_Complete=1&_ipg=200&LH_Sold=1&LH_PrefLoc=2&rt=nc&LH_BIN=1"
    print(url, file=sys.stderr)
    soup = get_data(url)
    productlist = parse(soup)
    productsdf = pd.DataFrame(productlist)
    productsdf.to_csv(sys.stdout, sep="\t", index=False)

def get_data(url):
    # 添加请求头模拟浏览器，避免被反爬拦截
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
    }
    r = requests.get(url, headers=headers)
    if r.status_code != 200:
        print('Failed to get data: ', r.status_code, file=sys.stderr)
        return BeautifulSoup('', 'html.parser')
    else:
        soup = BeautifulSoup(r.text, 'html.parser')
        print(soup.title.text, file=sys.stderr)
    return soup

def parse(soup):
    productlist = []
    # 匹配当前页面的结果容器选择器
    results = soup.find_all('div', {'class': 's-item__info clearfix'})
    for item in results:
        # 跳过顶部广告项
        title_elem = item.find('span', {'role':'heading'})
        if not title_elem or title_elem.text.strip() == "Shop on eBay":
            continue
        
        title = title_elem.text.strip()
        # 保留搜索词匹配逻辑，过滤低相关度结果
        terms = sys.argv[1].lower().split("+")
        query_words = title.lower().split()
        matched_terms = len(set(terms) & set(query_words))
        total_terms = len(set(terms))
        if total_terms > 0 and (matched_terms / total_terms) < 0.75:
            continue
        
        # 提取商品链接
        link_elem = item.find('a', {'class': 's-item__link'})
        link = link_elem['href'] if link_elem else None
        
        # 提取售出价格，兼容非数字格式
        price_elem = item.find('span', {'class':'s-item__price'})
        sold_price = None
        if price_elem:
            price_text = price_elem.text.strip().replace('£', '').replace(',', '')
            try:
                sold_price = float(price_text)
            except ValueError:
                sold_price = price_text
        
        # 提取购买地区，去除前缀
        location_elem = item.find('span', {'class': 's-item__location'})
        buyer_location = location_elem.text.strip().replace('from ', '') if location_elem else None
        
        # 提取售出日期，修正层级路径
        sold_date = None
        sold_tag = item.find('div', {'class': 's-item__title--tag'})
        if sold_tag:
            completed_block = sold_tag.find('span', {'class': 's-item__title--tagblock__COMPLETED'})
            if completed_block:
                date_elem = completed_block.find('span', {'class':'POSITIVE'})
                sold_date = date_elem.text.strip() if date_elem else None
        
        # 收集有效数据
        if title and sold_price:
            product = {
                'title': title,
                'sold_date': sold_date,
                'sold_price': sold_price,
                'buyer_location': buyer_location,
                'link': link
            }
            productlist.append(product)
    
    return productlist

if __name__ == '__main__':
    main()

关键修改说明 #

1. 反爬适配：添加User-Agent请求头模拟浏览器，避免被eBay拦截
2. URL修正：移除原URL中多余的+1参数，确保搜索词正确传递
3. 字段提取优化：
   • 售出日期：通过s-item__title--tag→s-item__title--tagblock__COMPLETED→POSITIVE的层级路径准确提取
   • 售出价格：增加异常处理，兼容特殊格式的价格文本
   • 购买地区：自动去除"from "前缀，直接提取地区名称
   • 商品链接：直接获取目标标签的href属性
4. 鲁棒性增强：对所有字段添加空值判断，避免单个字段缺失导致脚本崩溃
5. 过滤逻辑优化：提前跳过广告项，减少无效数据处理

内容的提问来源于stack exchange，提问作者O.rka