Scrapy Shell执行fetch获取URL时出现Spider事件循环错误求助

问题重现 #

在虚拟环境安装Scrapy后，使用Scrapy Shell执行fetch('https://www.digikala.com/')，返回以下错误：

>>> fetch('https://www.digikala.com/')
2023-01-30 14:45:55 [scrapy.core.engine] INFO: Spider opened
2023-01-30 14:45:55 [filelock] DEBUG: Attempting to acquire lock 140718217880064 on /home/ariyan/.cache/python-tldextract/3.10.6.final__venv__63b271__tldextract-3.4.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-01-30 14:45:55 [filelock] DEBUG: Lock 140718217880064 acquired on /home/ariyan/.cache/python-tldextract/3.10.6.final__venv__63b271__tldextract-3.4.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-01-30 14:45:55 [filelock] DEBUG: Attempting to release lock 140718217880064 on /home/ariyan/.cache/python-tldextract/3.10.6.final__venv__63b271__tldextract-3.4.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-01-30 14:45:55 [filelock] DEBUG: Lock 140718217880064 released on /home/ariyan/.cache/python-tldextract/3.10.6.final__venv__63b271__tldextract-3.4.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-01-30 14:45:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.digikala.com/robots.txt> (referer: None)
2023-01-30 14:45:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.digikala.com/> (referer: None)
>>> 2023-01-30 14:45:55 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.digikala.com/> (referer: None)
Traceback (most recent call last):
  File "/home/ariyan/venv/lib/python3.10/site-packages/twisted/internet/defer.py", line 892, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/home/ariyan/venv/lib/python3.10/site-packages/scrapy/utils/defer.py", line 285, in f
    return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
  File "/home/ariyan/venv/lib/python3.10/site-packages/scrapy/utils/defer.py", line 272, in deferred_from_coro
    event_loop = get_asyncio_event_loop_policy().get_event_loop()
  File "/usr/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-1 (start)'.
2023-01-30 14:45:55 [py.warnings] WARNING: /home/ariyan/venv/lib/python3.10/site-packages/twisted/internet/defer.py:892: RuntimeWarning: coroutine 'SpiderMiddlewareManager.scrape_response.<locals>.process_callback_output' was never awaited
  current.result = callback(  # type: ignore[misc]

错误原因 #

这是Scrapy版本与Python 3.10+的asyncio事件循环机制不兼容导致的，Scrapy Shell在异步回调线程中无法找到当前事件循环，进而触发报错。

解决方案 #

方案1：降级Scrapy到兼容版本

安装Scrapy 2.8.x版本（该版本对Python 3.10的兼容性更稳定）：

pip install scrapy==2.8.0

重新启动Scrapy Shell后执行fetch命令即可正常使用。

方案2：手动在Scrapy Shell中设置事件循环

如果不想降级版本，进入Scrapy Shell后先执行以下代码，再调用fetch：

import asyncio
asyncio.set_event_loop(asyncio.new_event_loop())
fetch('https://www.digikala.com/')

方案3：检查并修复依赖冲突

确保虚拟环境中Twisted版本与Scrapy兼容，可先卸载现有Twisted和Scrapy，再重新安装指定版本：

pip uninstall scrapy twisted -y
pip install scrapy==2.8.0

内容的提问来源于stack exchange，提问作者arian eyvazi