使用Django(Daphne)与Scrapy结合时爬虫阻塞问题求助

当不将daphne加入INSTALLED_APPS时，Scrapy爬虫运行完全正常；但启用Daphne后，爬虫在完成初始请求后卡住，始终无法进入parse方法。项目因依赖Django Channels必须使用Daphne。

项目结构 #

daphne_scrapy/
├── daphne_scrapy
│   ├── __init__.py
│   ├── asgi.py
│   ├── settings.py
│   ├── spiders.py
│   ├── urls.py
│   ├── views.py
│   └── wsgi.py
├── db.sqlite3
└── manage.py

settings.py修改内容 #

仅添加daphne和配置ASGI应用：

INSTALLED_APPS = [
    'daphne',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

WSGI_APPLICATION = 'daphne_scrapy.wsgi.application'
ASGI_APPLICATION = 'daphne_scrapy.asgi.application'

spider.py代码 #

注：需初始化Django以使用项目模型

import scrapy
from scrapy.crawler import CrawlerProcess
import django, os

os.environ.setdefault('DJANGO_SETTINGS_MODULE','settings')
django.setup()

class MySpider(scrapy.Spider):
    name = 'my_spider'
    start_urls = ['https://jsonplaceholder.typicode.com/todos/1']

    def parse(self, response):
        data = response.json()
        print('in parse method')
        yield data

def run_spider():
    process = CrawlerProcess()
    process.crawl(MySpider)
    process.start()

run_spider()

爬虫运行结果 #

运行后卡在以下输出阶段：

...

2023-04-24 03:23:33 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2023-04-24 03:23:33 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2023-04-24 03:23:33 [scrapy.core.engine] INFO: Spider opened
2023-04-24 03:23:34 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-04-24 03:23:34 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2023-04-24 03:23:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://jsonplaceholder.typicode.com/todos/1> (referer: None)
2023-04-24 03:23:34 [asyncio] DEBUG: Using selector: KqueueSelector

问题原因 #

Daphne作为ASGI服务器，会在Django初始化阶段修改全局的asyncio事件循环配置。而Scrapy底层依赖Twisted框架，当Twisted与被Daphne修改过的asyncio事件循环交互时，会出现调度冲突，导致Scrapy的回调方法（如parse）无法被正常触发，最终表现为爬虫卡住。

解决方法 #

方法1：启动Scrapy前重置asyncio事件循环 #

修改spider.py，在初始化CrawlerProcess前重置事件循环，同时修正Django配置路径的错误：

import scrapy
from scrapy.crawler import CrawlerProcess
import django, os
import asyncio

# 修正Django配置路径，确保能正确加载
os.environ.setdefault('DJANGO_SETTINGS_MODULE','daphne_scrapy.settings')
django.setup()

# 重置asyncio事件循环，避免与Daphne的配置冲突
asyncio.set_event_loop(asyncio.new_event_loop())

class MySpider(scrapy.Spider):
    name = 'my_spider'
    start_urls = ['https://jsonplaceholder.typicode.com/todos/1']

    def parse(self, response):
        data = response.json()
        print('in parse method')
        yield data

def run_spider():
    process = CrawlerProcess()
    process.crawl(MySpider)
    process.start()

run_spider()

方法2：使用Twisted Reactor直接启动爬虫 #

绕过asyncio事件循环的冲突，直接使用Twisted的Reactor来运行Scrapy：

import scrapy
from scrapy.crawler import CrawlerRunner
from twisted.internet import reactor
import django, os

os.environ.setdefault('DJANGO_SETTINGS_MODULE','daphne_scrapy.settings')
django.setup()

class MySpider(scrapy.Spider):
    name = 'my_spider'
    start_urls = ['https://jsonplaceholder.typicode.com/todos/1']

    def parse(self, response):
        data = response.json()
        print('in parse method')
        yield data

def run_spider():
    runner = CrawlerRunner()
    deferred = runner.crawl(MySpider)
    # 爬虫结束后停止Reactor
    deferred.addBoth(lambda _: reactor.stop())
    reactor.run()

run_spider()

方法3：强制Scrapy使用Twisted原生选择器 #

通过环境变量指定Scrapy使用Twisted的SelectReactor，避免使用asyncio相关的选择器：

import scrapy
from scrapy.crawler import CrawlerProcess
import django, os

# 强制使用Twisted原生选择器，不依赖asyncio
os.environ['TWISTED_REACTOR'] = 'twisted.internet.selectreactor.SelectReactor'

os.environ.setdefault('DJANGO_SETTINGS_MODULE','daphne_scrapy.settings')
django.setup()

class MySpider(scrapy.Spider):
    name = 'my_spider'
    start_urls = ['https://jsonplaceholder.typicode.com/todos/1']

    def parse(self, response):
        data = response.json()
        print('in parse method')
        yield data

def run_spider():
    process = CrawlerProcess()
    process.crawl(MySpider)
    process.start()

run_spider()

内容的提问来源于stack exchange，提问作者kizii