Scrapy与asyncio集成异常：使用CrawlerRunner时程序挂起

大家好，我最近在尝试把Scrapy和asyncio结合使用时碰到了一个棘手的问题，想请各位帮忙分析下。

先给大家看一下我的Spider代码：

import logging

from scrapy import Spider, Request, settings

from dummy  import settings
from dummy.items import DummyItem

LOGGER = logging.getLogger(__name__)


class DummySpider(Spider):
    name = 'DummySpider'

    def start_requests(self) -> Request:
        LOGGER.info("!!!!!!!!!!!!!!!!!!!!!!!!!! run start_requests")
        from twisted.internet import reactor
        LOGGER.info(f'reactor_class: {reactor.__class__}')
        LOGGER.info(f'scrapy_twisted_reactor: {settings.TWISTED_REACTOR}')

        yield Request('https://google.com', dont_filter=True)

    async def parse(self, _, **kwargs) -> DummyItem:
        LOGGER.info("!!!!!!!!!!!!!!!!!!!!!!!!!! run parse")
        item = DummyItem()
        item['id'] = 42
        yield item

当我直接用Scrapy命令运行这个Spider时，一切都正常，日志能完整输出，程序也能顺利结束：

2024-12-03 13:54:19 [scrapy.core.engine] INFO: Spider opened
2024-12-03 13:54:19 [dummy.spiders.dummy] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!! run start_requests
2024-12-03 13:54:19 [dummy.spiders.dummy] INFO: reactor_class: <class 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'>
2024-12-03 13:54:19 [dummy.spiders.dummy] INFO: scrapy_twisted_reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2024-12-03 13:54:19 [dummy.spiders.dummy] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!! run parse
2024-12-03 13:54:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.google.com/>
{'id': 42}
2024-12-03 13:54:19 [scrapy.core.engine] INFO: Closing spider (finished)

但当我改用CrawlerRunner结合asyncio来运行时，程序直接就挂起了——没有输出parse函数的日志，也不会继续执行，就停在那里一动不动。我的测试代码如下：

from scrapy.utils.reactor import install_reactor
install_reactor('twisted.internet.asyncioreactor.AsyncioSelectorReactor')
import asyncio
import logging

from scrapy.crawler import CrawlerRunner
from scrapy.utils.defer import deferred_to_future
from twisted.internet import reactor

from dummy.spiders.dummy import DummySpider

LOGGER = logging.getLogger(__name__)

class TaskError(Exception):
    pass


async def run_scraper() -> None:
    crawler_runner = CrawlerRunner()
    deferred = crawler_runner.crawl(DummySpider)
    print(deferred)
    future = deferred_to_future(deferred)
    print(future)
    print(reactor.__class__)
    await deferred_to_future(deferred)

if __name__ == "__main__":
    asyncio.run(run_scraper())

运行后只输出了这些内容，之后就彻底卡住了：

<Deferred at 0x71cf0c6c3e90>
<Future pending cb=[Deferred.asFuture.<locals>.checkCancel() at .venv/lib/python3.12/site-packages/twisted/internet/defer.py:1214]>
<class 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'>

我已经确认了当前使用的reactor和直接运行Scrapy时一致，都是AsyncioSelectorReactor，但就是找不到程序挂起的原因。有没有大佬能帮我诊断下问题，或者给个可行的解决办法？

备注：内容来源于stack exchange，提问作者Luis Sieira