如何为Scrapy特定Spider追加中间件并保留settings.py配置

场景描述 #

• 在settings.py中已为所有Spider启用了部分下载中间件
• 需要为某一特定Spider追加额外中间件，同时保留settings.py中的原有配置

全局配置示例 #

settings.py中的下载中间件配置如下：

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.CustomDownloaderMiddleware1': 543,
    'myproject.middlewares.CustomDownloaderMiddleware2': 544,
}

遇到的问题 #

直接在Spider的custom_settings中设置新中间件会忽略全局配置，尝试了两种实现方式，但新增的MyAdditionalMiddleware均未生效：

方式一：在__init__方法中修改配置 #

class MySpider(Spider):
    name = 'my_spider'
    
    def __init__(self, *args, **kwargs):
        super(MySpider, self).__init__(*args, **kwargs)
        settings = get_project_settings()

        # 获取现有中间件
        middlewares = settings.get('DOWNLOADER_MIDDLEWARES', {})
        
        # 添加额外中间件
        middlewares['myproject.middlewares.MyAdditionalMiddleware'] = 550
    
        # 应用到Spider配置
        self.custom_settings = {
            'DOWNLOADER_MIDDLEWARES': middlewares
        }
    
    def start_requests(self):
        # Spider逻辑
        pass

方式二：使用from_crawler方法修改配置 #

from scrapy.spiders import Spider
from scrapy.utils.project import get_project_settings
    
class MySpider(Spider):
    name = 'my_spider'
    
    @classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        spider = super(MySpider, cls).from_crawler(crawler, *args, **kwargs)
    
        # 获取全局配置
        settings = crawler.settings
    
        # 获取现有中间件并复制
        middlewares = settings.get('DOWNLOADER_MIDDLEWARES', {}).copy()
    
        # 添加额外中间件
        middlewares['myproject.middlewares.MyAdditionalMiddleware'] = 550
    
        # 更新当前Spider实例的配置
        spider.custom_settings = {
            'DOWNLOADER_MIDDLEWARES': middlewares
        }
    
        return spider
    
    def start_requests(self):
        # Spider逻辑
        pass

说明 #

上述两种实现代码由ChatGPT生成

内容的提问来源于stack exchange，提问作者Umair Ayub