Scrapy-Selenium爬虫切换页面触发KeyError: 'driver'的解决方案咨询

问题根源 #

你翻页时用了response.follow()生成普通Scrapy请求，这种请求不会经过Selenium中间件处理，所以response.meta['driver']拿不到浏览器驱动实例，直接触发KeyError。

修复步骤 #

1. 把翻页请求替换为SeleniumRequest #

所有页面请求都要通过Selenium驱动加载，把原来的response.follow()改成SeleniumRequest，确保每一页都能拿到driver实例。

2. 优化页码管理（避免多线程冲突） #

别用类变量page_number，改用response.meta传递页码，防止多爬虫实例运行时页码混乱。

3. 修正settings里的驱动路径 #

which()函数是从系统PATH找可执行文件，直接传绝对路径会返回None，导致驱动找不到，直接写绝对路径即可（用原始字符串避免转义问题）。

完整修正后的爬虫代码 #

import scrapy
from scrapy import Selector
from scrapy_selenium import SeleniumRequest

class TestSpider(scrapy.Spider):
    name = 'test'

    def start_requests(self):
        # 初始化页码到meta
        yield SeleniumRequest(
            url='https://barreau-montpellier.com/annuaire-professionnel/?cn-s=',
            callback=self.parse,
            meta={'page_number': 1}
        )

    def parse(self, response):
        driver = response.meta['driver']
        r = Selector(text=driver.page_source)

        details = r.xpath("//div[@class='cn-entry cn-background-gradient']")
        for detail in details:
            email = detail.xpath(".//span[@class='email cn-email-address']//a//@href").get()
            try:
                email = email.replace("mailto:", "")
            except:
                email = ''
            
            n1 = detail.xpath(".//span[@class='given-name']//text()").get() or ''
            n2 = detail.xpath(".//span[@class='family-name']//text()").get() or ''
            name = n1 + n2
            
            telephone = detail.xpath(".//span[@class='tel cn-phone-number cn-phone-number-type-workphone']//a//text()").get()
            fax = detail.xpath(".//span[@class='tel cn-phone-number cn-phone-number-type-workfax']//a//text()").get()

            street = detail.xpath(".//span[@class='adr cn-address']//span[@class='street-address notranslate']//text()").get() or ''
            locality = detail.xpath(".//span[@class='adr cn-address']//span[@class='locality']//text()").get() or ''
            code = detail.xpath(".//span[@class='adr cn-address']//span[@class='postal-code']//text()").get() or ''
            address = street + locality + code
            
            yield {
                'name': name,
                'mail': email,
                'telephone': telephone,
                'Fax': fax,
                'address': address
            }
        
        page_number = response.meta['page_number']
        if page_number <= 155:
            next_page = f'https://barreau-montpellier.com/annuaire-professionnel/pg/{page_number}/?cn-s'
            yield SeleniumRequest(
                url=next_page,
                callback=self.parse,
                meta={'page_number': page_number + 1}
            )

修正后的settings.py配置 #

SELENIUM_DRIVER_NAME = 'chrome'
# 用原始字符串避免转义问题
SELENIUM_DRIVER_EXECUTABLE_PATH = r'C:\Program Files (x86)\chromedriver.exe'
SELENIUM_DRIVER_ARGUMENTS=['--headless']  

DOWNLOADER_MIDDLEWARES = {
     'scrapy_selenium.SeleniumMiddleware': 800
}

内容的提问来源于stack exchange，提问作者Amen Aziz