Scrapy中yield Request未在每次迭代调用parse函数的问题

在Scrapy爬虫开发中，我通过start_requests方法从Excel读取车牌数据，赋值给全局变量plate_num_xlsx，循环生成请求并调用parse方法。预期每次迭代的parse都能使用当前的plate_num_xlsx值与解析结果对比，但实际运行时，所有parse调用都只拿到最后一次赋值的数值。

原始核心代码片段：

def start_requests(self):
    df=pd.read_excel('data.xlsx')
    columnA_values=df['PLATE']
    for row in columnA_values:
        global  plate_num_xlsx
        plate_num_xlsx=row
        print("+",plate_num_xlsx)
        base_url =f"https://dvlaregistrations.dvla.gov.uk/search/results.html?search={plate_num_xlsx}&action=index&pricefrom=0&priceto=&prefixmatches=&currentmatches=&limitprefix=&limitcurrent=&limitauction=&searched=true&openoption=&language=en&prefix2=Search&super=&super_pricefrom=&super_priceto="
        url=base_url
        yield scrapy.Request(url,callback=self.parse)

完整原始代码：

import scrapy
from scrapy.crawler import CrawlerProcess
import pandas as pd

itemList=[]
class plateScraper(scrapy.Spider):
    name = 'scrapePlate'
    allowed_domains = ['dvlaregistrations.dvla.gov.uk']

    def start_requests(self):
        df=pd.read_excel('data.xlsx')
        columnA_values=df['PLATE']
        for row in columnA_values:
            global  plate_num_xlsx
            plate_num_xlsx=row
            print("+",plate_num_xlsx)
            base_url =f"https://dvlaregistrations.dvla.gov.uk/search/results.html?search={plate_num_xlsx}&action=index&pricefrom=0&priceto=&prefixmatches=&currentmatches=&limitprefix=&limitcurrent=&limitauction=&searched=true&openoption=&language=en&prefix2=Search&super=&super_pricefrom=&super_priceto="
            url=base_url
            yield scrapy.Request(url,callback=self.parse)

    def parse(self, response):

        for row in response.css('div.resultsstrip'):
            plate = row.css('a::text').get()
            price = row.css('p::text').get()
            a = plate.replace(" ", "").strip()
            print(plate_num_xlsx,a,a == plate_num_xlsx)
            if plate_num_xlsx==plate.replace(" ","").strip():
                item= {"plate": plate.strip(), "price": price.strip()}
                itemList.append(item)
                yield  item
            else:
                item = {"plate": plate_num_xlsx, "price": "-"}
                itemList.append(item)
                yield item

        with pd.ExcelWriter('output_res.xlsx', mode='r+',if_sheet_exists='overlay') as writer:
            df_output = pd.DataFrame(itemList)
            df_output.to_excel(writer, sheet_name='result', index=False, header=True)

process = CrawlerProcess()
process.crawl(plateScraper)
process.start()

问题原因 #

Scrapy的请求是异步执行的：start_requests会快速完成所有循环，将所有请求加入任务队列，此时全局变量plate_num_xlsx已经被覆盖为最后一个车牌值。当parse方法实际执行时，读取的是已经更新后的全局变量，因此所有请求的parse都只会使用最后一次赋值的数值。

解决方案 #

不要用全局变量传递数据，而是利用Scrapy Request对象的meta参数携带当前迭代的车牌值。meta是请求的附加数据字典，可在response.meta中取出对应值，确保每个请求的parse都能拿到对应迭代的车牌数据。同时优化Excel写入逻辑，避免重复写入。

修改后的完整代码 #

import scrapy
from scrapy.crawler import CrawlerProcess
import pandas as pd

class plateScraper(scrapy.Spider):
    name = 'scrapePlate'
    allowed_domains = ['dvlaregistrations.dvla.gov.uk']
    
    # 用实例变量存储结果，替代全局itemList
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.itemList = []

    def start_requests(self):
        df = pd.read_excel('data.xlsx')
        columnA_values = df['PLATE']
        for row in columnA_values:
            plate_num = row
            print("+", plate_num)
            base_url = f"https://dvlaregistrations.dvla.gov.uk/search/results.html?search={plate_num}&action=index&pricefrom=0&priceto=&prefixmatches=&currentmatches=&limitprefix=&limitcurrent=&limitauction=&searched=true&openoption=&language=en&prefix2=Search&super=&super_pricefrom=&super_priceto="
            # 将当前车牌值放入meta参数
            yield scrapy.Request(url=base_url, callback=self.parse, meta={'plate_num': plate_num})

    def parse(self, response):
        # 从response.meta取出当前请求对应的车牌值
        plate_num_xlsx = response.meta['plate_num']
        
        for row in response.css('div.resultsstrip'):
            plate = row.css('a::text').get()
            price = row.css('p::text').get()
            if not plate:
                continue
            a = plate.replace(" ", "").strip()
            print(plate_num_xlsx, a, a == plate_num_xlsx)
            
            if plate_num_xlsx == a:
                item = {"plate": plate.strip(), "price": price.strip()}
            else:
                item = {"plate": plate_num_xlsx, "price": "-"}
            
            self.itemList.append(item)
            yield item

    # 爬虫结束时统一写入Excel，避免重复IO操作
    def closed(self, reason):
        with pd.ExcelWriter('output_res.xlsx', mode='w') as writer:
            df_output = pd.DataFrame(self.itemList)
            df_output.to_excel(writer, sheet_name='result', index=False, header=True)

process = CrawlerProcess()
process.crawl(plateScraper)
process.start()

关键修改点 #

1. 移除全局变量：删除全局的plate_num_xlsx和itemList，改用爬虫类的实例变量self.itemList存储结果，避免线程安全问题。
2. 使用meta传递数据：在start_requests中生成Request时，通过meta={'plate_num': plate_num}将当前车牌值附加到请求中；在parse中通过response.meta['plate_num']取出对应值。
3. 优化Excel写入：将Excel写入逻辑移到爬虫的closed方法中，爬虫结束后统一写入，减少文件IO次数，提升效率。

内容的提问来源于stack exchange，提问作者xlmaster