Scrapy多函数传递Item异常：返回空值或重复数据求助

问题现象 #

• 在parse_played_seasons函数中执行yield item时，返回的Item是空值（seasons_stats为空列表）
• 若将yield item移至parse_season函数中，则会生成重复的Item
• 目标是获取球员各赛季的完整统计数据

现有代码 #

class TransfersSpider(scrapy.Spider):
    name = "transfers"
    start_urls = 'https://www.transfermarkt.com/kevin-de-bruyne/profil/spieler/88755'

    def parse(self, response): 
        item = PlayerTransfersItem()
        info_table = response.css('div[class="info-table info-table--right-space "]') or  response.css('div[class="large-6 large-pull-6 small-12 columns spielerdatenundfakten"]')
        item["name"] = response.xpath('//h1/text()').get().strip() + " " + response.xpath('//h1//following-sibling::strong/text()').get(default = "")   

        stats_url = response.url.replace('profil', 'leistungsdaten') # 用于获取球员所有参赛赛季的URL
        yield scrapy.Request(stats_url, callback= self.parse_played_seasons, cb_kwargs={"item": item})

    def parse_played_seasons(self, response, item): # 获取球员所有参赛赛季
        item["seasons_stats"] = list()
        seasons = response.css('div[class="inline-select"] > select[name="saison"] >option::attr(value)').getall() # 获取所有赛季
        for season in seasons:  # 逐个解析赛季数据
            url = f"{response.url}/plus/0?saison={season}" 
            yield scrapy.Request(url, callback=self.parse_season, cb_kwargs= {"item": item, "season": season})
        yield item # 此处返回空值
     
    def parse_season(self, response, item, season):
        tables = response.css('div[class="box"] > div[class="responsive-table"]')
        total_table =  tables[0].css('tr> td::text').get()
        if "Total" in total_table: # 存在总计行的表格
            appearances_total_table = tables[0].css('tr> td[class="zentriert"]::text').get()
            goals_total_table = tables[0].css('tr> td[class="zentriert"]::text')[1].get()
            assists_total_table = tables[0].css('tr> td[class="zentriert"]::text')[2].get()
            season_stats = { season:{"total_goals": goals_total_table,"total_assists" : assists_total_table, "minutes_played": minutes_total_table,
                                     "appearances": appearances_total_table}}
            item["seasons_stats"].append(season_stats)

问题根源 #

1. 异步请求导致提前返回空Item：Scrapy的请求是异步执行的，parse_played_seasons里循环生成多个赛季请求后立刻yield item，此时所有parse_season的请求还未执行，seasons_stats列表是空的，所以返回的Item没有赛季数据。
2. 重复yield同一个Item对象：如果在parse_season里yield item，每个赛季请求完成都会触发一次yield，而所有请求共享同一个Item对象，每次yield时Item已经包含之前赛季的数据，所以会生成多条重复且数据越来越多的Item。

解决方法 #

核心思路是等所有赛季数据都爬取完成后，再统一yield Item，可以通过跟踪已处理赛季的数量实现：

修改后的代码 #

class TransfersSpider(scrapy.Spider):
    name = "transfers"
    start_urls = ['https://www.transfermarkt.com/kevin-de-bruyne/profil/spieler/88755']

    def parse(self, response): 
        item = PlayerTransfersItem()
        info_table = response.css('div[class="info-table info-table--right-space "]') or  response.css('div[class="large-6 large-pull-6 small-12 columns spielerdatenundfakten"]')
        item["name"] = response.xpath('//h1/text()').get().strip() + " " + response.xpath('//h1//following-sibling::strong/text()').get(default = "")   

        stats_url = response.url.replace('profil', 'leistungsdaten')
        yield scrapy.Request(stats_url, callback= self.parse_played_seasons, cb_kwargs={"item": item})

    def parse_played_seasons(self, response, item):
        item["seasons_stats"] = list()
        seasons = response.css('div[class="inline-select"] > select[name="saison"] >option::attr(value)').getall()
        total_seasons = len(seasons)
        
        # 没有赛季数据直接返回
        if total_seasons == 0:
            yield item
            return
            
        for season in seasons:
            url = f"{response.url}/plus/0?saison={season}" 
            yield scrapy.Request(
                url, 
                callback=self.parse_season, 
                cb_kwargs={
                    "item": item, 
                    "season": season, 
                    "total_seasons": total_seasons,
                    "processed": 0  # 已处理赛季数初始为0
                }
            )
     
    def parse_season(self, response, item, season, total_seasons, processed):
        tables = response.css('div[class="box"] > div[class="responsive-table"]')
        total_table = tables[0].css('tr> td::text').get()
        if "Total" in total_table:
            appearances_total_table = tables[0].css('tr> td[class="zentriert"]::text').get()
            goals_total_table = tables[0].css('tr> td[class="zentriert"]::text')[1].get()
            assists_total_table = tables[0].css('tr> td[class="zentriert"]::text')[2].get()
            # 补充获取出场时间的逻辑（原代码未定义minutes_total_table）
            minutes_total_table = tables[0].css('tr> td[class="zentriert"]::text')[3].get(default="0")
            
            season_stats = {
                season: {
                    "total_goals": goals_total_table,
                    "total_assists": assists_total_table, 
                    "minutes_played": minutes_total_table,
                    "appearances": appearances_total_table
                }
            }
            item["seasons_stats"].append(season_stats)
        
        # 已处理赛季数+1
        processed += 1
        # 所有赛季处理完成后，才yield完整的Item
        if processed == total_seasons:
            yield item

补充说明 #

• 修复了原代码中minutes_total_table未定义的问题，根据页面结构补充了选择器（实际需根据目标页面调整位置）
• 通过传递total_seasons和processed参数，跟踪所有赛季的处理进度，确保只有当所有赛季数据都收集完成时才返回Item
• 增加了无赛季数据的边界处理，避免空转

内容的提问来源于stack exchange，提问作者Baraa Zaid