如何按顺序运行多个Scrapy爬虫？解决ReactorNotRestartable错误

我编写了一段用于按顺序启动Scrapy爬虫的代码：

def start_crawling():
    spiders = [spdier1, spider2]
    runner = CrawlerProcess()

    @defer.inlineCallbacks
    def crawl():
        for spider in spiders:
            yield runner.crawl(spider, (task.id, True, None))
        reactor.stop()

    while True:
        a = time.time()
        crawl()
        reactor.run()
        print("crawl time: " + str(time.time() - a))

if name == "main": 
   start_crawling()

运行时触发错误：twisted.internet.error.ReactorNotRestartable，错误日志如下：

2022-12-27 21:10:39 [scrapy.core.engine] INFO: Spider opened
2022-12-27 21:10:39 [scrapy.extensions.logstats] INFO: Crawled   pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-12-27 21:10:39 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
Traceback (most recent call last):
File "//./scraper.py", line 49, in <module>
start_crawling()
File "//./scraper.py", line 44, in start_crawling
reactor.run()
File "/usr/local/lib/python3.10/site-packages/twisted/internet/base.py", line 1317, in run
self.startRunning(installSignalHandlers=installSignalHandlers)
File "/usr/local/lib/python3.10/site-packages/twisted/internet/base.py", line 1299, in 
ReactorBase.startRunning(cast(ReactorBase, self))
File "/usr/local/lib/python3.10/site-packages/twisted/internet/base.py", line 843, in 
raise error.ReactorNotRestartable()
twisted.internet.error.ReactorNotRestartable

请问该如何修复这个错误？

错误原因 #

Twisted的reactor实例是不可重启的，第一次调用reactor.stop()后，该实例就无法再次通过reactor.run()启动。你的代码用while True循环反复重启reactor，直接触发了这个错误。

推荐修复方式：基于Twisted延迟任务实现循环 #

把循环逻辑放到reactor内部，用task.deferLater安排下一次爬虫执行，全程保持一个reactor实例运行，符合异步编程模型：

from twisted.internet import reactor, task
from scrapy.crawler import CrawlerProcess
from scrapy.utils.defer import deferInlineCallbacks
import time

def start_crawling():
    spiders = [spider1, spider2]  # 修正原代码拼写错误：spdier1 → spider1

    def run_crawl():
        # 每次执行创建新的CrawlerProcess实例，避免复用导致的状态问题
        runner = CrawlerProcess()

        @deferInlineCallbacks
        def crawl():
            start_time = time.time()
            for spider in spiders:
                yield runner.crawl(spider, (task.id, True, None))
            yield runner.start()  # 用start()替代手动stop，更符合Scrapy流程
            print(f"crawl time: {time.time() - start_time}")
            # 爬虫完成后延迟指定时间执行下一轮（示例间隔60秒，可按需调整）
            task.deferLater(reactor, 60, run_crawl)

        crawl()

    # 启动第一次爬虫任务
    run_crawl()
    reactor.run()

if __name__ == "__main__":  # 修正原代码语法错误：name → __name__，main → __main__
   start_crawling()

额外修正点 #

• 原代码中spdier1是拼写错误，需改为spider1
• 原代码if name == "main"不符合Python入口文件规范，需改为if __name__ == "__main__"
• 用runner.start()替代手动调用reactor.stop()，Scrapy会在所有爬虫完成后自动处理reactor状态

内容的提问来源于stack exchange，提问作者Tetraborat1