Selenium+BeautifulSoup亚马逊多页数据采集函数复用方案咨询

问题解答 #

函数复用可行性 #

完全可以复用你定义的采集函数，既然你已经验证过第二页的DOM结构和首页完全一致，只需要对现有函数做少量调整就能适配多页采集场景。

现有函数调整建议 #

原函数依赖全局soup变量、页码写死为1、缺少异常兼容，调整后的代码如下：

from datetime import date
import pandas as pd

def data_collection(soup, page_num):
    title = soup.find_all(name = "span", class_ = "a-size-base-plus a-color-base a-text-normal")
    all_specs = [specs.getText().strip() for specs in title]
    # 增加异常判断，避免特殊格式标题导致拆分报错
    brands = []
    model = []
    specifications = []
    for item in all_specs:
        try:
            brands.append(item.split(' ', 1)[0])
            phone_part = item.split(')')[0].split('(')
            model.append(phone_part[0].strip())
            specifications.append(phone_part[1].strip() if len(phone_part)>=2 else '')
        except:
            # 格式异常条目填充占位值，避免采集中断
            brands.append('')
            model.append('')
            specifications.append('')

    s_price_obj = soup.find_all(name = "span", class_ = "a-price-whole")
    selling_price = [price.getText().strip() for price in s_price_obj]
    # 动态长度对齐，避免部分页商品数量不足导致数组错位
    if len(selling_price) < len(all_specs):
        selling_price += ['']*(len(all_specs)-len(selling_price))

    review_obj = soup.find_all(name = "span", class_ = "a-icon-alt")
    review = [ratings.getText().strip() for ratings in review_obj]
    review = review[:len(all_specs)]
    if len(review) < len(all_specs):
        review += ['']*(len(all_specs)-len(review))

    quantity_obj = soup.find_all(name = "span", class_ = "a-size-base")
    quantity_sold = [items.getText().strip() for items in quantity_obj]
    quantity_sold = quantity_sold[:len(all_specs)]
    if len(quantity_sold) < len(all_specs):
        quantity_sold += ['']*(len(all_specs)-len(quantity_sold))

    page_number = [str(page_num)]*len(all_specs)
    current_date = str(date.today())
    Date = [current_date]*len(all_specs)

    # 转为按行存储结构，方便后续转表导出
    page_data = []
    for i in range(len(all_specs)):
        page_data.append({
            '品牌': brands[i],
            '型号': model[i],
            '参数': specifications[i],
            '售价': selling_price[i],
            '评分': review[i],
            '销量': quantity_sold[i],
            '页码': page_number[i],
            '采集日期': Date[i]
        })
    return page_data

多页采集+CSV导出实现流程 #

total_data = []
for page in range(1,7):
    # 此处替换为你自己的页面加载逻辑：首页直接生成soup，第2-6页调用click()跳转后等待加载完成再生成soup
    page_data = data_collection(soup, page)
    total_data.extend(page_data)

# 导出为CSV文件
df = pd.DataFrame(total_data)
df.to_csv('亚马逊手机采集数据.csv', index=False, encoding='utf-8-sig')

额外优化建议 #

• 用Selenium的WebDriverWait显式等待目标元素加载完成再生成soup，避免页面未加载全导致空数据
• 每次跳转页面后加1-3秒的随机延迟，降低被亚马逊反爬拦截的概率
• 增加页面跳转、采集的异常重试逻辑，单页采集失败可以重试2-3次再跳过，避免整个流程中断

内容的提问来源于stack exchange，提问作者Akhil Nair