Scrapy：触发重复过滤器时终止爬虫进程的技术问询

这确实是个爬虫开发中挺常见的棘手问题——不想硬编码易变的总页码，又得避免爬虫在重复内容上无限空转。我整理了几个实用的方案，你可以根据自己的场景选择：

方法1：自定义重复过滤器，触发重复时终止爬虫 #

Scrapy自带的RFPDupeFilter只会过滤重复请求，但不会主动终止爬虫。我们可以继承它，添加“检测到重复就关闭爬虫”的逻辑：

首先在项目的middlewares.py里写自定义过滤器：

from scrapy.dupefilters import RFPDupeFilter
from scrapy.crawler import Crawler

class TerminatingDupeFilter(RFPDupeFilter):
    def __init__(self, path=None, debug=False):
        super().__init__(path, debug)
        self.crawler = None

    @classmethod
    def from_crawler(cls, crawler: Crawler):
        instance = cls(crawler.settings.get('DUPEFILTER_DEBUG'))
        instance.crawler = crawler
        return instance

    def request_seen(self, request):
        # 先调用父类方法判断是否重复
        seen = super().request_seen(request)
        if seen:
            # 检测到重复请求，直接关闭爬虫
            self.crawler.engine.close_spider(
                self.crawler.spider, 
                reason='Duplicate request detected, terminating spider'
            )
        return seen

然后在settings.py里替换默认的重复过滤器：

DUPEFILTER_CLASS = '你的项目名.middlewares.TerminatingDupeFilter'

这个方案适合请求URL重复的场景——比如网站把/page/26的请求重定向到/page/1，此时/page/26的URL会被识别为重复，触发终止。

方法2：跟踪已爬取条目，检测全重复页面后终止 #

如果网站超页后返回的是第一页内容，但URL还是/page/26（不会被请求重复过滤器拦截），那更可靠的方式是跟踪已爬取的条目唯一标识，当新页面的所有条目都已存在时终止爬虫：

import scrapy
from scrapy.exceptions import CloseSpider

class MySpider(scrapy.Spider):
    name = 'my_spider'
    start_urls = ['https://example.com/page/1']
    # 存储已爬取的条目唯一ID（比如item的data-id、详情页URL等）
    seen_item_ids = set()

    def parse(self, response):
        items = response.css('.item-container')
        current_page_item_ids = set()

        for item in items:
            # 提取条目唯一标识，这里用data-id举例
            item_id = item.css('::attr(data-id)').get()
            current_page_item_ids.add(item_id)
            
            if item_id not in self.seen_item_ids:
                self.seen_item_ids.add(item_id)
                # 处理并输出item
                yield {
                    'id': item_id,
                    'title': item.css('.title::text').get().strip(),
                    # 其他字段...
                }

        # 检查当前页面的所有条目是否都已爬过
        if current_page_item_ids.issubset(self.seen_item_ids):
            raise CloseSpider(reason='All items on this page are already scraped, terminating')

        # 构造下一页请求
        next_page_num = int(response.url.split('/')[-1]) + 1
        next_page_url = f'https://example.com/page/{next_page_num}'
        yield scrapy.Request(next_page_url, callback=self.parse)

这个方案更精准，因为它直接基于内容重复判断，不受URL变化的影响，推荐优先考虑这种方式。

方法3：设置重复阈值，达到次数后终止 #

如果担心单次重复就终止太激进（比如偶尔的网络波动导致重复请求），可以统计重复次数，达到阈值再终止：

修改自定义重复过滤器：

from scrapy.dupefilters import RFPDupeFilter
from scrapy.crawler import Crawler

class ThresholdTerminatingDupeFilter(RFPDupeFilter):
    def __init__(self, path=None, debug=False):
        super().__init__(path, debug)
        self.crawler = None
        self.duplicate_count = 0
        # 默认重复3次就终止
        self.duplicate_threshold = 3

    @classmethod
    def from_crawler(cls, crawler: Crawler):
        instance = cls(crawler.settings.get('DUPEFILTER_DEBUG'))
        instance.crawler = crawler
        # 可以从settings里配置阈值
        instance.duplicate_threshold = crawler.settings.getint('DUPLICATE_THRESHOLD', 3)
        return instance

    def request_seen(self, request):
        seen = super().request_seen(request)
        if seen:
            self.duplicate_count += 1
            if self.duplicate_count >= self.duplicate_threshold:
                self.crawler.engine.close_spider(
                    self.crawler.spider, 
                    reason=f'Duplicate requests reached threshold ({self.duplicate_threshold}), terminating'
                )
        return seen

然后在settings.py里配置：

DUPEFILTER_CLASS = '你的项目名.middlewares.ThresholdTerminatingDupeFilter'
DUPLICATE_THRESHOLD = 3

内容的提问来源于stack exchange，提问作者gunesevitan