Scrapy执行fetch(url)命令时遇RuntimeError问题求助

环境信息 #

• 系统：Ubuntu 22.04
• 运行环境：Miniconda
• Scrapy版本：2.7.1
• Python版本：3.10.6

操作步骤 #

scrapy startproject worldometers
cd worldometers
scrapy shell
fetch("https://www.worldometers.info/world-population/population-by-country/")

错误信息 #

File "/home/miniconda3/envs/main/lib/python3.10/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/home/miniconda3/envs/main/lib/python3.10/site-packages/scrapy/utils/defer.py", line 285, in f
    return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
  File "/home/miniconda3/envs/main/lib/python3.10/site-packages/scrapy/utils/defer.py", line 272, in deferred_from_coro
    event_loop = get_asyncio_event_loop_policy().get_event_loop()
  File "/home/miniconda3/envs/main/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-1 (start)'.
2022-11-12 21:08:55 [py.warnings] WARNING: /home/miniconda3/envs/main/lib/python3.10/site-packages/twisted/internet/defer.py:857: RuntimeWarning: coroutine 'SpiderMiddlewareManager.scrape_response.<locals>.process_callback_output' was never awaited
  current.result = callback(  # type: ignore[misc]

修复建议 #

方案1：降级Scrapy版本到2.6.x #

该错误源于Scrapy 2.7.x与Python 3.10在异步事件循环处理上的兼容性问题，降级到稳定的2.6版本可直接解决：

pip install scrapy==2.6.3

方案2：指定异步事件循环策略 #

在Scrapy项目的settings.py中添加以下配置，强制使用Twisted的异步 reactor：

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

添加后重启Scrapy shell，重新执行fetch命令。

方案3：升级依赖包 #

更新Twisted和asyncio相关依赖，确保版本兼容：

pip install --upgrade twisted asyncio

内容的提问来源于stack exchange，提问作者Mario Jaya