为何标准Scrapy可爬取API，而Selenium版本返回401错误？

我正在使用Selenium驱动的Scrapy爬虫爬取网站，以避免被检测为机器人。该网站包含多个可调用的API，无需下载完整页面即可获取数据。但以下代码始终返回401错误：

import logging

from scrapy import Request, Spider, signals
from scrapy.crawler import Crawler, CrawlerProcess
from scrapy.http import HtmlResponse
from selenium.webdriver import Firefox, FirefoxOptions

import scrape.constants as cs


class FlashscoreMiddleware:
    @classmethod
    def from_crawler(cls, crawler: Crawler):
        middleware = cls()
        crawler.signals.connect(middleware.spider_opened, signals.spider_opened)
        crawler.signals.connect(middleware.spider_closed, signals.spider_closed)
        return middleware

    def spider_opened(self, *_) -> None:
        self.driver = Firefox()

    def spider_closed(self, *_) -> None:
        self.driver.quit()

    def process_request(self, request: Request, **_) -> None:
        self.driver.get(request.url)
        return HtmlResponse(
            request.url,
            body=self.driver.page_source,
            encoding="utf-8",
            request=request,
        )


class FlashscoreSpider(Spider):
    name = "flashscore"
    custom_settings = {
        "DOWNLOADER_MIDDLEWARES": {"selenium_scraper.FlashscoreMiddleware": 400},
        "REQUEST_FINGERPRINTER_IMPLEMENTATION": "2.7",
        "LOG_LEVEL": logging.ERROR,
    }

    def start_requests(self):
        yield Request(
            url="https://global.flashscore.ninja/5/x/feed/df_st_1_WKM03Vff",
            headers={
                "X-Fsign": "SW9D1eZo",
                "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36",
            },
            callback=self.parse,
        )

    def parse(self, response):
        if response.text.startswith("SE÷Match"):
            print(f"Successfully returned {response.url}")
        else:
            print("Unsuccessful")


if __name__ == "__main__":
    process = CrawlerProcess()
    process.crawl(FlashscoreSpider)
    process.start()

然而，若使用标准Scrapy爬虫则一切正常：

import logging

from scrapy import Request, Spider
from scrapy.crawler import CrawlerProcess


class FlashscoreSpider(Spider):
    name = "flashscore"
    custom_settings = {
        "REQUEST_FINGERPRINTER_IMPLEMENTATION": "2.7",
        "LOG_LEVEL": logging.ERROR,
    }

    def start_requests(self):
        yield Request(
            url="https://global.flashscore.ninja/5/x/feed/df_st_1_WKM03Vff",
            headers={"X-Fsign": "SW9D1eZo"},
            callback=self.parse,
        )

    def parse(self, response):
        if response.text.startswith("SE÷Match"):
            print(f"Successfully returned {response.url}")
        else:
            print("Unsuccessful")


if __name__ == "__main__":
    process = CrawlerProcess()
    process.crawl(FlashscoreSpider)
    process.start()

为什么Selenium版本无法返回正确结果？

• 自定义请求头未传递：你在Scrapy Request里设置的X-Fsign和自定义User-Agent并没有被Selenium的浏览器请求带上。driver.get(request.url)只会用浏览器默认的请求头发起请求，缺少API要求的X-Fsign验证字段，直接导致401未授权错误。
• 请求特征不匹配：Selenium驱动的浏览器自带的请求头、指纹特征和标准Scrapy请求不一致，也可能被网站的反爬机制拦截。

解决步骤 #

1. 给Selenium请求添加自定义请求头
   Selenium本身不支持直接修改请求头，可通过Chrome DevTools Protocol（CDP）实现（推荐改用Chrome驱动）。示例修改中间件：

   from selenium.webdriver import Chrome, ChromeOptions
   
   class FlashscoreMiddleware:
       # ... 其他方法不变 ...
       def spider_opened(self, *_) -> None:
           options = ChromeOptions()
           # 可选：启用无头模式
           # options.add_argument("--headless=new")
           self.driver = Chrome(options=options)
   
       def process_request(self, request: Request, **_) -> None:
           # 通过CDP注入自定义请求头
           headers = {}
           for key, value in request.headers.items():
               headers[key.decode()] = value.decode()
           self.driver.execute_cdp_cmd('Network.setExtraHTTPHeaders', {'headers': headers})
           # 启用网络监听
           self.driver.execute_cdp_cmd('Network.enable', {})
           self.driver.get(request.url)
           return HtmlResponse(
               request.url,
               body=self.driver.page_source,
               encoding="utf-8",
               request=request,
           )
   

2. 调整Scrapy配置确保中间件接管请求
   在Spider的custom_settings里禁用无关的默认下载中间件，避免干扰：

   custom_settings = {
       "DOWNLOADER_MIDDLEWARES": {
           "scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": None,
           "selenium_scraper.FlashscoreMiddleware": 400,
       },
       "REQUEST_FINGERPRINTER_IMPLEMENTATION": "2.7",
       "LOG_LEVEL": logging.ERROR,
   }
   

3. 验证请求头是否正确传递
   运行Selenium爬虫时打开浏览器开发者工具（可添加options.add_argument("--auto-open-devtools-for-tabs")），在Network面板查看请求头，确认X-Fsign和User-Agent与标准Scrapy请求一致。

内容的提问来源于stack exchange，提问作者Jossy