如何用Python爬取亚马逊全层级分类URL直到最末级子分类

核心逻辑为封装递归爬取函数，逐层级提取子分类URL，直到页面无更多子分类即为末级分类，写入结果文件。同时增加了URL拼接、去重、异常处理逻辑，避免重复爬取和请求报错中断。

from bs4 import BeautifulSoup
import requests

# 固定配置
HEADERS = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0", "Accept-Encoding":"gzip, deflate", "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "DNT":"1","Connection":"close", "Upgrade-Insecure-Requests":"1"}
BASE_URL = "https://www.amazon.com"
# 已爬取URL集合，避免重复请求
crawled_urls = set()

def crawl_category(current_url, output_file):
    # 去重判断，已经爬过的URL直接跳过
    if current_url in crawled_urls:
        return
    crawled_urls.add(current_url)
    
    # 请求当前分类页，增加异常处理
    try:
        r = requests.get(current_url, headers=HEADERS, timeout=10)
        r.raise_for_status()
    except Exception as e:
        print(f"请求失败 {current_url}: {str(e)}")
        return
    
    soup = BeautifulSoup(r.content, "lxml")
    # 提取当前分类下的子分类链接
    sub_links = soup.find_all('a', {'class': 'a-link-normal s-navigation-item'})
    
    # 没有子分类，说明是末级分类，写入结果
    if not sub_links:
        # 提取当前分类信息和商品数量
        try:
            counts = soup.find('div', {'class' :'a-section a-spacing-small a-spacing-top-small'}).text.strip()
            cat = soup.find('div', {'class' :'nav-search-facade'}).text.strip()
            output_file.write(f"{cat},{counts},{current_url}\n")
        except Exception as e:
            print(f"提取末级分类信息失败 {current_url}: {str(e)}")
        return
    
    # 有子分类，递归遍历所有子分类
    for link in sub_links:
        sub_href = link.get('href')
        # 拼接完整URL
        if sub_href.startswith('/'):
            sub_full_url = BASE_URL + sub_href
        else:
            sub_full_url = sub_href
        crawl_category(sub_full_url, output_file)

if __name__ == "__main__":
    # 入口分类URL（图书主分类）
    start_url = "https://www.amazon.com/s?i=stripbooks&bbn=283155&rh=n%3A283155%2Cp_n_feature_browse-bin%3A2656022011&dc&qid=1636126733&ref=sr_ex_n_1"
    # 打开结果文件，提前写入表头
    with open("parseddata.csv", "w", encoding="utf-8-sig") as f:
        f.write("分类名称,商品数量,分类URL\n")
        # 启动递归爬取
        crawl_category(start_url, f)
    print("爬取完成")

关键改动说明 #

• 封装crawl_category递归函数，每次处理单个分类URL，自动判断是否存在子分类，存在就继续向下爬取
• 新增crawled_urls集合做去重，避免相同URL被重复请求导致死循环
• 处理相对路径拼接，将提取到的短链接补全为亚马逊完整可访问URL
• 增加异常捕获逻辑，请求失败、信息提取失败不会中断整体爬取流程
• 统一使用with语法管理文件操作，避免资源泄露，新增UTF-8编码兼容中文和特殊字符

内容的提问来源于stack exchange，提问作者Satti22