Scrapy中POST请求获取变体产品详情及CSV导出问题求助

问题分析与修复方案 #

一、POST请求失败的核心问题及修复

你的代码里POST请求无法正常返回数据，主要有以下几个错误：

1. 硬编码URL参数：AJAX URL里的id_product=19107和token=d41d8cd98f00b204e9800998ecf8427e是固定值，应该替换为从页面提取的动态值。
2. Payload格式不匹配：用json.dumps()生成JSON字符串，但请求头设为application/x-www-form-urlencoded，两者格式冲突，需改用表单编码格式。
3. Payload参数错误：controller字段错误设为产品标题，实际应为固定值product；且只获取了第一个变体的group值，未遍历所有变体选项。
4. 未传递基础产品数据：没有把原始item通过meta传递给parse_variants，无法合并变体数据与基础产品信息。

二、变体数据存入CSV的处理逻辑

要让每个变体成为CSV的单独一行，需在parse_variants中将响应返回的变体数据（如价格、EAN码等）合并到原始item的副本中，再yield该合并后的item，Scrapy的CSV导出器会自动处理为多行。

三、修改后的完整代码

import scrapy
import os
import json
from slugify import slugify
from urllib.parse import urlencode

class GpSpider(scrapy.Spider):
    name = 'gp'
    start_urls = ['https://goldpet.pt/3-cao']
    user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36'

    def parse(self, response):
        products = response.css('h2.h3.product-title>a::attr(href)').extract()
        for product in products:
            yield scrapy.Request(product, callback=self.parse_products, headers={'User-Agent': self.user_agent})

    def parse_products(self, response):
        # 提取基础产品数据
        item = {}
        item['Title'] = response.css('h1.h1.product-title::text').get().strip()
        item['Descrição'] = '\n'.join([p.strip() for p in response.css('div.product-description>p::text').extract() if p.strip()])
        item['Marca'] = response.css('img.img.img-thumbnail.manufacturer-logo::attr(alt)').get() or 'No brand'
        item['Idade'] = response.xpath('//dt[text()="Idade"]/following-sibling::dd/text()').get() or ''
        item['Porte'] = response.xpath('//dt[text()="Porte"]/following-sibling::dd/text()').get() or ''
        item['Características'] = response.xpath('//dt[text()="Características"]/following-sibling::dd/text()').get() or ''
        item['Gama'] = response.xpath('//dt[text()="Gama"]/following-sibling::dd/text()').get() or ''
        item['Alimento'] = response.xpath('//dt[text()="Alimento"]/following-sibling::dd/text()').get() or ''
        item['product_url'] = response.url

        # 处理面包屑分类
        breadcrumbs = list(filter(None, map(str.strip, response.css('li[itemprop=itemListElement]>a>span::text').extract())))
        item['category'] = breadcrumbs[0] if len(breadcrumbs)>=1 else ''
        item['sub_category1'] = breadcrumbs[1] if len(breadcrumbs)>=2 else ''
        item['sub_category2'] = breadcrumbs[2] if len(breadcrumbs)>=3 else ''

        # 处理图片
        product_imgs = response.css('img.js-qv-product-cover::attr(src)').getall()
        if product_imgs:
            # 主图
            main_img = product_imgs[0]
            ext = main_img.split('?')[0].rsplit('.', 1)[-1]
            filename = slugify(item['Title']) + '_1.' + ext
            item['Photo_0'] = filename
            item['Photo_Path0'] = os.path.join('product images', 'images', item['Marca'], filename)
            
            # 其他图片（最多10张）
            for i in range(1, 11):
                item[f'Photo_{i}'] = ''
                item[f'Photo_Path_{i}'] = ''
            
            for idx, img in enumerate(product_imgs[1:], start=1):
                if idx > 10:
                    break
                ext = img.split('?')[0].rsplit('.', 1)[-1]
                filename = slugify(item['Title']) + f'_{idx+1}.{ext}'
                item[f'Photo_{idx}'] = filename
                item[f'Photo_Path_{idx}'] = os.path.join('product images', 'images', item['Marca'], filename)

        # 处理变体
        variants_container = response.css('div.products-variants')
        if variants_container:
            # 提取所有变体的group值（数量选项对应的value）
            group_values = response.css('li.input-container.float-xs-left label input.input-radio::attr(value)').getall()
            token = response.css('div.product-actions form#add-to-cart-or-refresh input::attr(value)').get()
            product_id = response.css('input#product_page_product_id::attr(value)').get()
            customization_id = response.css('input#product_customization_id::attr(value)').get() or '0'

            for group in group_values:
                # 动态构造AJAX URL
                ajax_params = {
                    'controller': 'product',
                    'token': token,
                    'id_product': product_id,
                    'id_customization': customization_id,
                    f'group[8]': group,
                    'qty': '1'
                }
                ajax_url = f'https://goldpet.pt/index.php?{urlencode(ajax_params)}'

                # 构造表单格式的Payload
                payload = {
                    'controller': 'product',
                    'token': token,
                    'id_product': product_id,
                    'id_customization': customization_id,
                    f'group[8]': group,
                    'qty': '1'
                }
                # 通过meta传递原始item副本
                yield scrapy.Request(
                    ajax_url,
                    callback=self.parse_variants,
                    method="POST",
                    body=urlencode(payload),
                    headers={
                        'Content-Type': 'application/x-www-form-urlencoded',
                        'User-Agent': self.user_agent,
                        'X-Requested-With': 'XMLHttpRequest'  # 模拟AJAX请求标识
                    },
                    meta={'base_item': item.copy()}
                )
        else:
            # 无变体的产品，直接提取默认价格和ean13
            item['Price'] = response.css('div.current-price>span::text').get().replace('\xa0€','').strip() if response.css('div.current-price>span::text').get() else ''
            item['ean13'] = response.xpath('//dt[text()="ean13"]/following-sibling::dd/text()').get() or ''
            yield item

    def parse_variants(self, response):
        base_item = response.meta['base_item']
        variant_data = json.loads(response.text)
        
        # 合并变体数据到基础item
        base_item['Price'] = variant_data['product']['price'].replace('\xa0€','').strip()
        base_item['ean13'] = next((attr['value'] for attr in variant_data['product']['attributes'] if attr['name'] == 'ean13'), '')
        # 获取当前变体的数量标签
        group_value = response.request.body.decode().split('group%5B8%5D=')[1].split('&')[0]
        variant_label = response.css(f'li.input-container.float-xs-left label input.input-radio[value="{group_value}"]+span.radio-label::text').get().strip()
        base_item['Quantidade'] = variant_label
        
        # 输出变体item，每个变体会作为CSV的一行
        yield base_item

关键修改说明

1. 动态构造请求：用页面提取的token和product_id生成URL与Payload，避免硬编码。
2. 修正Payload格式：用urlencode()将字典转为表单编码字符串，匹配请求头格式。
3. 传递基础数据：通过meta传递原始item副本，确保变体数据能与基础产品信息合并。
4. 遍历所有变体：获取所有group值，为每个变体发送独立POST请求。
5. 合并数据输出：将变体返回的价格、EAN码等信息合并到基础item，yield后自动存入CSV单独行。

运行与导出CSV

执行以下命令启动爬虫并导出CSV：

scrapy crawl gp -o products.csv

内容的提问来源于stack exchange，提问作者Chiedozie Agwu