使用Celery调度Scrapy遇ReactorNotRestartable及反应器不匹配问题

在使用Celery调度Scrapy爬虫时，遇到了ReactorNotRestartable错误，尝试改用twisted.internet.asyncioreactor.AsyncioSelectorReactor替代默认反应器后，又出现已安装反应器与请求的反应器不匹配的异常。

问题代码 #

from scrapy.utils.log import configure_logging
from multiprocessing import Process, Queue

def run_spider(spider, domain=None, check=None):
    def f(q):
        try:
            configure_logging()
            runner = CrawlerRunner(get_project_settings())
            deferred = runner.crawl(spider, domain=domain, check=check)
            deferred.addBoth(lambda _: reactor.stop())
            reactor = AsyncioSelectorReactor()
            reactor.run()
            q.put(None)
        except Exception as e:
            print("EXCEPTION!")
            q.put(e)
    
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    result = q.get()
    p.join()

    if result is not None:
        raise result

报错信息 #

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1697, in _inlineCallbacks
    result = context.run(gen.send, result)
  File "/code/scrapy_parsing/scripts/run_spider.py", line 203, in crawl
    yield runner.crawl(spider)
  File "/usr/local/lib/python3.10/site-packages/scrapy/crawler.py", line 232, in crawl
    crawler = self.create_crawler(crawler_or_spidercls)
  File "/usr/local/lib/python3.10/site-packages/scrapy/crawler.py", line 266, in create_crawler
    return self._create_crawler(crawler_or_spidercls)
  File "/usr/local/lib/python3.10/site-packages/scrapy/crawler.py", line 271, in _create_crawler
    return Crawler(spidercls, self.settings)
  File "/usr/local/lib/python3.10/site-packages/scrapy/crawler.py", line 103, in __init__
    verify_installed_reactor(reactor_class)
  File "/usr/local/lib/python3.10/site-packages/scrapy/utils/reactor.py", line 138, in verify_installed_reactor
    raise Exception(msg)
Exception: The installed reactor (twisted.internet.epollreactor.EPollReactor) does not match the requested one (twisted.internet.asyncioreactor.AsyncioSelectorReactor)

问题原因及解决方法 #

问题核心是反应器初始化时机太晚：创建CrawlerRunner时Scrapy已经加载了默认反应器，后续手动初始化的AsyncioSelectorReactor无法覆盖已加载的实例，导致匹配失败。

修复方案1：代码中提前安装反应器

必须在任何Scrapy相关代码执行前，先安装指定的反应器：

from scrapy.utils.log import configure_logging
from multiprocessing import Process, Queue
from twisted.internet import reactor
from twisted.internet.asyncioreactor import AsyncioSelectorReactor
from scrapy.crawler import CrawlerRunner
from scrapy.utils.project import get_project_settings

def run_spider(spider, domain=None, check=None):
    def f(q):
        try:
            # 优先级最高：先安装指定反应器
            reactor.install(AsyncioSelectorReactor())
            configure_logging()
            runner = CrawlerRunner(get_project_settings())
            deferred = runner.crawl(spider, domain=domain, check=check)
            deferred.addBoth(lambda _: reactor.stop())
            reactor.run()
            q.put(None)
        except Exception as e:
            print("EXCEPTION!")
            q.put(e)
    
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    result = q.get()
    p.join()

    if result is not None:
        raise result

修复方案2：通过Scrapy配置全局指定反应器

在项目的settings.py中添加配置，让Scrapy启动时自动加载指定反应器，无需在代码中重复设置：

# settings.py
TWISTED_REACTOR = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'

内容的提问来源于stack exchange，提问作者gzm