You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

Scrapy爬虫中断时spider_closed信号未触发的问题排查

avatar
AI 小北
2026-6-16
Scrapy爬虫Ctrl-C中断时spider_closed信号未触发的问题

问题描述

自定义了Scrapy爬虫LinkExtractorSpider、Pipeline并连接了spider_closed信号，爬虫正常结束时，callback和Pipeline的spider_closed方法日志均正常显示；但使用Ctrl-C中断脚本时，程序虽正常退出，上述日志却都未出现。

相关代码 

class LinkExtractorSpider(scrapy.spiders.CrawlSpider):
    name = 'LinkExtractor'

    rules = [
        scrapy.spiders.Rule(
            scrapy.linkextractors.LinkExtractor(),
            callback='produce_url',
            follow=True,
        )
    ]

    def start_requests(self):
        yield scrapy.Request(url=self.settings['START_URL'])

    def produce_url(self, response):
        yield {'url': response.url}


class Pipeline:
    def process_item(self, item, spider):
        logging.info(f'Check: {item}')

    def spider_closed(self, spider, reason):
        logging.info(f'Spider closed because: {reason}')
        logging.info('Cleanup code would go here')

    @classmethod
    def from_crawler(cls, crawler):
        pipeline = Pipeline()
        crawler.signals.connect(pipeline.spider_closed, signal=scrapy.signals.spider_closed)
        return pipeline


class Crawler:
    def __init__(self):
        os.environ['SCRAPY_SETTINGS_MODULE'] = 'settings'
        scrapy.utils.log.configure_logging()
        self.settings = scrapy.utils.project.get_project_settings()
        self.runner = scrapy.crawler.CrawlerRunner(self.settings)

    def should_run_again(self):
        return False

    def callback(self, results):
        logging.info("Maybe I want to run again, maybe not")
        if self.should_run_again():
            self.runner.crawl(LinkExtractorSpider).addCallback(self.callback)
        else:
            twisted.internet.reactor.stop()

    def run(self, start_url):
        if not start_url.startswith('http'):
            start_url = 'https://' + start_url
        self.settings.set('START_URL', start_url)
        self.runner.crawl(LinkExtractorSpider).addCallback(self.callback)
        twisted.internet.reactor.run()

if __name__ == '__main__':
    Crawler().run('some_website.com')

Settings配置 

import logging

ITEM_PIPELINES = {
    "spider.Pipeline": 100,
}

LOG_LEVEL = logging.INFO
LOG_FORMAT = "%(name)s: [%(levelname)s] %(message)s"
LOG_STDOUT = True

ROBOTSTXT_OBEY = False
DOWNLOAD_TIMEOUT = 10
RETRY_ENABLED = False
DEPTH_LIMIT = 2

运行日志 

scrapy.addons: [INFO] Enabled addons:
[]
py.warnings: [WARNING] /Users/daniel/environments/poisonedparadox/lib/python3.10/site-packages/scrapy/utils/request.py:254: ScrapyDeprecationWarning: '2.6' is a deprecated value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting.

It is also the default value. In other words, it is normal to get this warning if you have not defined a value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. This is so for backward compatibility reasons, but it will change in a future version of Scrapy.

See the documentation of the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting for information on how to handle this deprecation.
  return cls(crawler)

scrapy.extensions.telnet: [INFO] Telnet Password: aaaaaaaaa
scrapy.middleware: [INFO] Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
scrapy.crawler: [INFO] Overridden settings:
{'DEPTH_LIMIT': 2,
 'DOWNLOAD_TIMEOUT': 10,
 'LOG_FORMAT': '%(name)s: [%(levelname)s] %(message)s',
 'LOG_LEVEL': 20,
 'LOG_STDOUT': True,
 'RETRY_ENABLED': False}
scrapy.middleware: [INFO] Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
scrapy.middleware: [INFO] Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
scrapy.middleware: [INFO] Enabled item pipelines:
['spider.Pipeline']
scrapy.core.engine: [INFO] Spider opened
scrapy.extensions.logstats: [INFO] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
scrapy.extensions.telnet: [INFO] Telnet console listening on 127.0.0.1:6023
root: [INFO] Check: {'url': 'https://www.google.com/intl/en/policies/privacy/'}
root: [INFO] Check: {'url': 'https://www.google.com/imghp?hl=en&tab=wi'}
root: [INFO] Check: {'url': 'https://www.google.com/intl/en/policies/terms/'}
root: [INFO] Check: {'url': 'https://www.google.com/preferences?hl=en'}
root: [INFO] Check: {'url': 'https://www.google.com/advanced_search?hl=en&authuser=0'}
root: [INFO] Check: {'url': 'https://smallbusiness.withgoogle.com/?subid=us-en-et-g-awa-a-g_hpbfoot1_1!o2&utm_source=google&utm_medium=ep&utm_campaign=google_hpbfooter&utm_content=google_hpbfooter&gmbsrc=us-en_US-et-gs-z-gmb-s-z-u~sb-g4sb_srvcs-u'}
root: [INFO] Check: {'url': 'https://www.google.com/maps?hl=en&tab=wl'}
root: [INFO] Check: {'url': 'https://ads.google.com/intl/en/home/'}
root: [INFO] Check: {'url': 'https://about.google/'}
root: [INFO] Check: {'url': 'https://www.google.com/history/optout?hl=en'}
root: [INFO] Check: {'url': 'https://news.google.com/home?tab=wn&hl=en-US&gl=US&ceid=US:en'}
root: [INFO] Check: {'url': 'https://about.google/products/?tab=wh'}
root: [INFO] Check: {'url': 'https://www.youtube.com/?tab=w1'}
root: [INFO] Check: {'url': 'https://accounts.google.com/v3/signin/identifier?continue=https%3A%2F%2Fwww.google.com%2F&ec=GAZAAQ&hl=en&ifkv=AcMMx-d2GgD-YTcHQJRQ98PqHE-zo-q3UYpsuA9APS1CY6G3mBsmxU_kgZuengZK0fVjecGRTCpIug&passive=true&flowName=WebLiteSignIn&flowEntry=ServiceLogin&dsh=S-1389799524%3A1730486180270110'}
root: [INFO] Check: {'url': 'https://www.google.com/advanced_image_search?hl=en&authuser=0'}
root: [INFO] Check: {'url': 'https://www.google.com/search?q=how+to+vote+in+the+us&hl=en&gl=us&stick=H4sIAAAAAAAAAHvE6Mgt8PLHPWEpi0lrTl5jNOISdM1JTS7JzM_zyC8PyQ_LL0kVkuXigAkKCUrxc_Hqp-sbGuaWFeRmZVTl8CxiFc3IL1coyVcoA6pWyMxTKMlIVSgtBgCjAj5eXAAAAA&utm_source=discover&utm_medium=promo&utm_campaign=13'}
root: [INFO] Check: {'url': 'https://policies.google.com/privacy?hl=en&gl=us'}
root: [INFO] Check: {'url': 'https://policies.google.com/terms'}
^Cscrapy.core.downloader.handlers.http11: [WARNING] Got data loss in https://play.google.com/store/games?hl=en&tab=w8. If you want to process broken responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False -- This message won't be shown in further requests
scrapy.core.downloader.handlers.http11: [WARNING] Got data loss in https://support.google.com/. If you want to process broken responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False -- This message won't be shown in further requests

解决方案

原因分析

按下Ctrl-C时系统发送SIGINT信号，Twisted reactor会直接停止，导致Scrapy跳过爬虫关闭流程，无法触发spider_closed信号和回调函数。

解决步骤

  1. 捕获SIGINT信号，优雅关闭爬虫
    在Crawler类中添加信号处理逻辑，捕获SIGINT后调用爬虫的close方法，而非直接停止reactor。

  2. 修改Crawler类代码
    调整run方法并添加信号处理：

    import signal
from twisted.internet import reactor

class Crawler:
    def __init__(self):
        os.environ['SCRAPY_SETTINGS_MODULE'] = 'settings'
        scrapy.utils.log.configure_logging()
        self.settings = scrapy.utils.project.get_project_settings()
        self.runner = scrapy.crawler.CrawlerRunner(self.settings)
        self.crawler = None

    def should_run_again(self):
        return False

    def callback(self, results):
        logging.info("Maybe I want to run again, maybe not")
        if self.should_run_again():
            self.runner.crawl(LinkExtractorSpider).addCallback(self.callback)
        else:
            reactor.stop()

    def handle_sigint(self, signum, frame):
        if self.crawler and self.crawler.engine.running:
            logging.info("Received SIGINT, closing spider gracefully...")
            self.crawler.close()

    def run(self, start_url):
        if not start_url.startswith('http'):
            start_url = 'https://' + start_url
        self.settings.set('START_URL', start_url)
        # 保存crawler实例
        self.crawler = self.runner.create_crawler(LinkExtractorSpider)
        self.runner.crawl(self.crawler).addCallback(self.callback)
        # 注册SIGINT信号处理
        signal.signal(signal.SIGINT, self.handle_sigint)
        reactor.run()

  3. 验证效果
    修改后按下Ctrl-C，Scrapy会完成爬虫关闭流程，触发spider_closed信号，Pipeline的spider_closed方法和回调函数日志将正常输出。

额外说明

  • 调用crawler.close()会让Scrapy执行完整的关闭逻辑，包括触发所有相关信号。
  • 避免直接调用reactor.stop()，否则会跳过爬虫关闭步骤。

内容的提问来源于stack exchange，提问作者August Vrubel

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
火山引擎 最新活动
全面开放
Doubao-Seedance-2.5全新模型升级

模型再升级,30秒超长叙事, 模态参考扩容

HOT
方舟Codingplan

模型自由，工具不限，最新支持 Deepseek-V4 系列、GLM-5.3 系列

HOT
方舟Agent plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列

HOT
新用户特惠专场

大模型19元起，Al应用9.9元畅享，新人首购爆款尽享优惠

最近更新时间：2026.06.16 13:52:33