Scrapy实现颜色与尺寸多组合商品变体爬取问题咨询

需要用Scrapy爬取指定商品的颜色与尺寸变体数据：

• 目标颜色：螳螂绿（Mantis Green）、线轴黄（Spool Yellow）
• 目标尺寸：6lb、8lb、10lb、15lb、20lb、30lb
  要获取每个变体组合（如「螳螂绿 6lb」）的标题、原价及特惠价，但现有代码无法生成所有颜色与尺寸的组合请求，导致爬取不到完整数据。

import scrapy
import re
from scrapy_splash import SplashRequest

class FishingRodsSpider(scrapy.Spider):
    name = "ana_rods_detailed"
    allowed_domains = ["anacondastores.com"]
    start_urls = ["https://www.anacondastores.com/fishing/fishing-line/braid-line/shimano-kairiki-8-braid-line-150-metre-spool/BP90140299"]

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse, args={'wait': 2})

    def parse(self, response):
        title = response.css('.pdp-title::text').get().strip()
        price = response.css('.product-info .price-was .amount::text').get().strip()
        club_price = response.css('.product-info .price-now .amount::text').get().strip()
        product_details = response.css('.product-details-list')
        base_url = "https://www.anacondastores.com"
        variant_style_pickers = response.css('.js-variant-style-picker')
        variant_size_pickers = response.css('.js-variant-size')
        style_data_urls = []
        for i, style_picker in enumerate(variant_style_pickers):
            style_data_url = style_picker.attrib.get('data-url')
            if i == 0 and style_data_url is None:
                for size_picker in variant_size_pickers:
                    size_data_url = size_picker.attrib.get('data-url')
                    if size_data_url is not None:
                        size_variant_url = base_url + size_data_url
                        # self.log("Size Variant: " + size_variant_url)
                    else:
                        self.log("Size data-url attribute is missing for a size picker.")
            else:
                if style_data_url is not None:
                    style_data_urls.append(base_url + style_data_url)
                    for size_picker in variant_size_pickers:
                        size_data_url = size_picker.attrib.get('data-url')
                        if size_data_url is not None:
                            size_variant_url = base_url + size_data_url
                            self.log("Size Variant: " + size_variant_url)
                else:
                    self.log("Style data-url attribute is missing for a style picker.")

修改后的代码会先筛选出目标颜色和尺寸的变体链接，再生成所有有效组合的请求，最后爬取每个变体的详细数据：

import scrapy
from scrapy_splash import SplashRequest

class FishingRodsSpider(scrapy.Spider):
    name = "ana_rods_detailed"
    allowed_domains = ["anacondastores.com"]
    start_urls = ["https://www.anacondastores.com/fishing/fishing-line/braid-line/shimano-kairiki-8-braid-line-150-metre-spool/BP90140299"]
    # 定义要爬取的目标颜色和尺寸
    TARGET_COLORS = ["Mantis Green", "Spool Yellow"]
    TARGET_SIZES = ["6lb", "8lb", "10lb", "15lb", "20lb", "30lb"]

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse_main_page, args={'wait': 2})

    def parse_main_page(self, response):
        base_url = "https://www.anacondastores.com"
        # 提取目标颜色对应的变体页面链接
        color_links = []
        for style_item in response.css('.js-variant-style-picker'):
            color_name = style_item.css('::attr(title)').get().strip()
            data_url = style_item.attrib.get('data-url')
            if color_name in self.TARGET_COLORS and data_url:
                color_links.append(base_url + data_url)
        
        # 对每个颜色页面发起请求，同时传递目标尺寸列表
        for color_url in color_links:
            yield SplashRequest(
                color_url, 
                self.parse_color_page, 
                args={'wait': 2}, 
                meta={'target_sizes': self.TARGET_SIZES}
            )

    def parse_color_page(self, response):
        base_url = "https://www.anacondastores.com"
        target_sizes = response.meta.get('target_sizes')
        # 获取当前页面的颜色名称
        current_color = response.css('.selected-variant-value::text').get().strip()
        
        # 提取当前颜色下的目标尺寸变体链接
        for size_item in response.css('.js-variant-size'):
            size_name = size_item.css('::text').get().strip()
            data_url = size_item.attrib.get('data-url')
            if size_name in target_sizes and data_url:
                variant_url = base_url + data_url
                # 发起变体页面请求，传递颜色和尺寸信息
                yield SplashRequest(
                    variant_url, 
                    self.parse_variant, 
                    args={'wait': 2}, 
                    meta={'color': current_color, 'size': size_name}
                )

    def parse_variant(self, response):
        # 获取传递的颜色和尺寸信息
        color = response.meta.get('color')
        size = response.meta.get('size')
        
        # 爬取变体的核心数据
        title = response.css('.pdp-title::text').get().strip()
        original_price = response.css('.product-info .price-was .amount::text').get()
        original_price = original_price.strip() if original_price else "无原价"
        sale_price = response.css('.product-info .price-now .amount::text').get()
        sale_price = sale_price.strip() if sale_price else "无特惠价"
        
        # 返回整理后的结果
        yield {
            'variant_name': f"{color} {size}",
            'title': title,
            'original_price': original_price,
            'sale_price': sale_price
        }

关键修改说明 #

1. 精准筛选目标变体：直接定义要爬取的颜色和尺寸列表，避免处理无关变体
2. 分阶段请求逻辑：先请求颜色页面，再在对应颜色页面提取目标尺寸链接，确保组合的有效性（避免网站不存在的无效组合）
3. 独立变体解析函数：专门处理单个变体页面的数据爬取，逻辑更清晰易维护
4. 元数据传递：通过meta参数传递颜色和尺寸信息，方便在最终结果中明确标识变体名称

爬取结果将包含以下所有组合的详细数据：

• 螳螂绿 6lb
• 螳螂绿 8lb
• 螳螂绿 10lb
• 螳螂绿 15lb
• 螳螂绿 20lb
• 螳螂绿 30lb
• 线轴黄 6lb
• 线轴黄 8lb
• 线轴黄 10lb
• 线轴黄 15lb
• 线轴黄 20lb
• 线轴黄 30lb

内容的提问来源于stack exchange，提问作者Patrick Flores