Scrapy+SeleniumBase爬虫出现未请求的chrome-extension URL异常

用Scrapy 2.11.2搭配SeleniumBase 4.28.5爬取https://www.atptour.com，碰到三个问题：

1. 没主动请求过chrome-extension相关地址，但日志里出现了这类请求，referer还是自身的扩展地址
2. 后续爬虫触发回调函数匹配异常，重装依赖也解决不了
3. 目标站存在重定向但返回200状态码，绕开了dont_redirect配置，重新发请求才能恢复正常

日志详情：

2024-10-21 17:43:47: [INFO] Spider opened
2024-10-21 17:43:47: [INFO] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-10-21 17:43:47: [INFO] Telnet console listening on 127.0.0.1:6027
2024-10-21 17:43:50: [DEBUG] Started executable: `/Users/philipjoss/miniconda3/envs/capra_production/lib/python3.11/site-packages/seleniumbase/drivers/uc_driver` in a child process with pid: 22177 using 0 to output -1
2024-10-21 17:43:51: [DEBUG] Crawled (200) <GET https://www.atptour.com/en/-/tournaments/calendar/tour> (referer: None)
2024-10-21 17:43:54: [DEBUG] Started executable: `/Users/philipjoss/miniconda3/envs/capra_production/lib/python3.11/site-packages/seleniumbase/drivers/uc_driver` in a child process with pid: 22180 using 0 to output -1
2024-10-21 17:43:55: [DEBUG] Crawled (200) <GET https://www.atptour.com/en/-/tournaments/calendar/challenger> (referer: None)
2024-10-21 17:43:55: [DEBUG] Crawled (200) <GET chrome-extension://neajdppkdcdipfabeoofebfddakdcjhd/audio.html> (referer: chrome-extension://neajdppkdcdipfabeoofebfddakdcjhd/audio.html)

注：相同依赖版本在另一台机器上运行正常。

1. chrome-extension请求的由来 #

• SeleniumBase的uc_driver（基于undetected-chromedriver）默认会加载内置扩展，或者本地Chrome残留的扩展被驱动复用，部分扩展（比如日志里的音频类扩展）会自动发起请求
• 如果指定了user_data_dir，该目录可能存在旧的扩展配置，导致驱动启动时加载了不必要的扩展

2. 回调匹配异常+重定向绕开配置的原因 #

• 中间件返回HtmlResponse时用了driver.current_url，如果目标站有前端JS重定向，这个地址已经是跳转后的，但Scrapy是按原始请求URL匹配回调的，直接导致回调关联失败
• dont_redirect只对Scrapy原生处理的HTTP重定向（3xx状态码）生效，前端JS跳转属于页面内行为，不受这个配置控制

解决chrome-extension请求问题 #

• 启动Driver时禁用所有扩展，添加extension_load_policy=2参数（2表示禁止加载任何扩展），修改中间件的Driver初始化代码：

self.driver = sb.Driver(
    uc=settings.get("UNDETECTABLE", None),
    headless=settings.get("HEADLESS", None),
    user_data_dir=settings.get("USER_DATA_DIR", None),
    extension_load_policy=2,  # 禁用所有扩展
    exclude_switches=["enable-automation"],  # 增强反检测，可选
)

• 如果用了user_data_dir，确保该目录是干净的，没有残留扩展；或者直接不指定，让驱动用临时目录

解决回调匹配异常+前端重定向问题 #

• 中间件返回HtmlResponse时，保留原始请求的URL，不要替换成driver.current_url，保证Scrapy能正确匹配回调：

return HtmlResponse(
    request.url,  # 用原始请求URL而非跳转后的地址
    body=self.driver.page_source,
    encoding="utf-8",
    request=request,
)

• 如果需要处理前端重定向，在回调函数里判断当前响应URL和原始请求URL是否一致，不一致则手动处理：

def _parse_calendar(self, response: HtmlResponse):
    if response.url != response.request.url:
        self.logger.warning(f"检测到前端跳转：{response.request.url} -> {response.url}")
        # 重新发起请求，添加dont_filter避免被去重
        yield Request(
            url=response.request.url,
            callback=self._parse_calendar,
            meta=dict(dont_redirect=True),
            dont_filter=True
        )
        return
    json_str = response.xpath("//body//text()").get()

• 可以配置Scrapy的DOWNLOAD_DELAY或开启AUTOTHROTTLE_ENABLED，降低请求频率，减少站点触发前端跳转的概率

爬虫代码 #

from scrapy import Request, Spider
from scrapy.http.response.html import HtmlResponse


class Production(Spider):

    name = "atp_production"

    start_urls = [
        "https://www.atptour.com/en/-/tournaments/calendar/tour",
        "https://www.atptour.com/en/-/tournaments/calendar/challenger",
    ]

    def start_requests(self):
        for url in self.start_urls:
            yield Request(
                url=url,
                callback=self._parse_calendar,
                meta=dict(dont_redirect=True),
            )

    def _parse_calendar(self, response: HtmlResponse):
        json_str = response.xpath("//body//text()").get()

中间件代码 #

class SeleniumBase:
    @classmethod
    def from_crawler(cls, crawler: Crawler):
        middleware = cls(crawler.settings)
        crawler.signals.connect(middleware.spider_closed, signals.spider_closed)
        
        return middleware

    def __init__(self, settings: dict[str, Any]) -> None:
        self.driver = sb.Driver(
            uc=settings.get("UNDETECTABLE", None),
            headless=settings.get("HEADLESS", None),
            user_data_dir=settings.get("USER_DATA_DIR", None),
        )

    def spider_closed(self, *_) -> None:
        self.driver.quit()

    def process_request(self, request: Request, spider: Spider) -> HtmlResponse:
        self.driver.get(request.url)

        return HtmlResponse(
            self.driver.current_url,
            body=self.driver.page_source,
            encoding="utf-8",
            request=request,
        )

内容的提问来源于stack exchange，提问作者Jossy