Scrapy多函数运行时遇NameError：scrapedate未定义

要把parse方法里的scrapedate传递到parse_race方法，核心是利用Scrapy的cb_kwargs参数在回调函数间传递数据，同时修正item实例化位置避免数据覆盖。具体操作如下：

1. 构建回调链传递scrapedate #

• 在parse方法调用response.follow时，通过cb_kwargs传入scrapedate
• 让parse_date、parse_meeting依次接收并传递该参数，最终传递到parse_race
• parse_race接收参数后，即可正常为DogItem的date字段赋值

2. 修正DogItem实例化位置 #

原代码中dog_item定义在循环外，会导致所有狗的数据被最后一条覆盖，需将dog_item移到for dog in dogs循环内部，确保每条狗对应独立的item对象。

修改后的完整代码 #

import scrapy
from datetime import datetime, timedelta
from dogscraper.items import DogItem

racedate = '2024-01-25'
days = 2
realdate = datetime.strptime(racedate, '%Y-%m-%d').date()
scrape_list = [(realdate - timedelta(days=x)).strftime('%Y-%m-%d') for x in range(days)]

class DogspiderSpider(scrapy.Spider):
    name = "dogspider"
    allowed_domains = ["www.thedogs.com.au"]
    start_urls = ["https://www.thedogs.com.au/racing/"+racedate]

    def parse(self, response):
        for scrapedate in scrape_list:
            next_dateurl = 'https://www.thedogs.com.au/racing/' + scrapedate
            # 传递scrapedate到parse_date
            yield response.follow(next_dateurl, callback=self.parse_date, cb_kwargs={'scrapedate': scrapedate})

    def parse_date(self, response, scrapedate):
        nswmeetings = response.css('table.meeting-grid')[0]
        nswmeetings = nswmeetings.css('td.meetings-venues__name')

        for meeting in nswmeetings:
            meeting_url = meeting.css('a::attr(href)').get()
            nextmeeting = 'https://www.thedogs.com.au' + meeting_url
            # 传递scrapedate到parse_meeting
            yield response.follow(nextmeeting, callback=self.parse_meeting, cb_kwargs={'scrapedate': scrapedate})

    def parse_meeting(self, response, scrapedate):
        races = response.css('a.race-box.race-box--result')
        for race in races:
            race_url = race.css('a.race-box.race-box--result::attr(href)').get()
            nextrace = 'https://www.thedogs.com.au' + race_url
            # 传递scrapedate到parse_race
            yield response.follow(nextrace, callback=self.parse_race, cb_kwargs={'scrapedate': scrapedate}) 

    def parse_race(self, response, scrapedate):
        dogs = response.css('tr.accordion__anchor.race-runner')
        
        for dog in dogs:               
            # 每条狗创建独立的item实例
            dog_item = DogItem()
            dog_item['date'] = scrapedate
            # 可在此添加其他字段的赋值逻辑
            yield dog_item

关键说明 #

• cb_kwargs是Scrapy 2.1+推荐的回调参数传递方式，相比旧的meta更直观且类型安全
• 确保每个回调函数都正确接收并传递scrapedate，形成完整的参数传递链
• 将dog_item放在循环内，避免不同狗的数据互相覆盖

内容的提问来源于stack exchange，提问作者jw32001