Scrapy爬取Galaxus网站：不同品类产品规格字段共用相同Xpath的解决方法咨询

Hey there! Let's break down how to solve both of your issues—capturing those dynamic product specifications and writing the scraped data back to your original Excel file.

1. Handling Dynamic Specification Fields #

The problem here is that different product categories use unique specification labels (like "Formfaktor" for SSDs and "Arbeitsspeichertyp" for RAM) but share the same XPath for their values. Instead of targeting individual fields directly, we can scrape all specification key-value pairs as a dictionary, then extract the fields you need (or keep all of them).

Here's how to adjust your parse method:

• First, grab all specification rows from the page
• For each row, extract the label (key) and corresponding value
• Store these in a dictionary, then add it to your scraped item

def parse(self, response):
    # Extract all specification rows
    spec_rows = response.xpath("//div[@data-test='productSpecifications']//tr")
    specs = {}
    for row in spec_rows:
        # Get the specification label (e.g., "Formfaktor")
        spec_key = row.xpath(".//td[@class='sc-18g78bs-2 fxQQoF']/text()").get(default='').strip()
        # Get the corresponding value
        spec_value = row.xpath(".//td[@class='sc-18g78bs-4 sxRfA']/text()").get(default='').strip()
        if spec_key:
            specs[spec_key] = spec_value

    # Add the Gtin to your item (critical for merging back to Excel later)
    gtin = response.meta['gtin']
    
    yield {
        'Gtin': gtin,
        'Titel': response.xpath(".//span[@class='jqo5ci-1 goteOY']/text()").get(),
        'Untertitel': response.xpath(".//span[@class='jqo5ci-2 beeFWi']/text()").get(),
        'Beschreibung': response.xpath("//div[@class='sc-1op7ol6-0 hYPLAr']/span/text()").get(),
        'Kategorie': response.xpath("(.//div[@class='breadcrumbView_withIcon__3mWwP']/a)[4]/text()").get(),
        'Produktetyp': response.xpath(".//span[@class='yip624-0 dpAcNY']/text()").get(),
        'Hersteller': response.xpath(".//h1[@class='jqo5ci-0 czhxQj']/strong/text()").get(),
        'Specifications': specs  # Keep all specs as a dictionary
    }

Bonus: Split Specifications into Individual Columns #

If you want each specification as its own column in your Excel file, you can use pandas.json_normalize() later when processing the data. This will automatically turn keys like "Formfaktor" and "Arbeitsspeichertyp" into separate columns.

2. Writing Results Back to the Original Excel File #

To merge your scraped data with the original Excel, we'll use Scrapy's spider_closed signal to trigger the merge once the crawl finishes. Here's the full updated spider code with this functionality:

import scrapy
from scrapy_splash import SplashRequest
from scrapy import signals
from scrapy.signalmanager import dispatcher
import pandas as pd
from galaxus.spiders.read_files import read_xlsx

base_url = "https://www.galaxus.ch/search?q={}"

class GtinSpider(scrapy.Spider):
    name = 'gtin'
    allowed_domains = ['www.galaxus.ch']
    script = '''
        function main(splash, args)
            splash:set_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36")
            splash.private_mode_enabled = false
            assert(splash:go(args.url))
            assert(splash:wait(5))
            item_select = assert(splash:select("div.panelLayout_mainContainer__11Jh_"))
            item_select:mouse_click()
            assert(splash:wait(5))
            see_more = assert(splash:select("[data-test='showMoreButton-specifications'] span"))
            see_more:mouse_click()
            assert(splash:wait(5))
            splash:set_viewport_full()
            return splash:html()
        end
    '''
    
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.scraped_items = []
        # Connect the spider_closed signal to our handler
        dispatcher.connect(self.merge_and_save_excel, signals.spider_closed)
    
    def start_requests(self):
        for gtin_value in read_xlsx():
            yield SplashRequest(
                url=base_url.format(gtin_value),
                callback=self.parse,
                endpoint='execute',
                args={'lua_source': self.script},
                meta={'gtin': gtin_value}  # Pass Gtin to parse method via meta
            )
    
    def parse(self, response):
        spec_rows = response.xpath("//div[@data-test='productSpecifications']//tr")
        specs = {}
        for row in spec_rows:
            spec_key = row.xpath(".//td[@class='sc-18g78bs-2 fxQQoF']/text()").get(default='').strip()
            spec_value = row.xpath(".//td[@class='sc-18g78bs-4 sxRfA']/text()").get(default='').strip()
            if spec_key:
                specs[spec_key] = spec_value

        item = {
            'Gtin': response.meta['gtin'],
            'Titel': response.xpath(".//span[@class='jqo5ci-1 goteOY']/text()").get(),
            'Untertitel': response.xpath(".//span[@class='jqo5ci-2 beeFWi']/text()").get(),
            'Beschreibung': response.xpath("//div[@class='sc-1op7ol6-0 hYPLAr']/span/text()").get(),
            'Kategorie': response.xpath("(.//div[@class='breadcrumbView_withIcon__3mWwP']/a)[4]/text()").get(),
            'Produktetyp': response.xpath(".//span[@class='yip624-0 dpAcNY']/text()").get(),
            'Hersteller': response.xpath(".//h1[@class='jqo5ci-0 czhxQj']/strong/text()").get(),
            'Specifications': specs
        }
        self.scraped_items.append(item)
        yield item
    
    def merge_and_save_excel(self, spider):
        # Read original Excel file
        df_original = pd.read_excel('externe_festplatte.xlsx')
        # Convert scraped items to DataFrame, expanding specs into columns
        df_scraped = pd.json_normalize(self.scraped_items)
        # Merge on Gtin to keep original data and add scraped columns
        df_merged = pd.merge(df_original, df_scraped, on='Gtin', how='left')
        # Write back to the original Excel file
        df_merged.to_excel('externe_festplatte.xlsx', index=False)
        self.logger.info(f"Successfully merged scraped data into externe_festplatte.xlsx")

Key Notes: #

• We pass the gtin value via meta in start_requests so we can include it in each scraped item—this is essential for matching rows in the original Excel.
• pd.json_normalize() expands the Specifications dictionary into individual columns, so you'll have columns like Specifications.Formfaktor and Specifications.Arbeitsspeichertyp automatically.
• The how='left' merge ensures we keep all rows from the original Excel, even if no data was scraped for a particular Gtin.

内容的提问来源于stack exchange，提问作者Raisul Islam