Scrapy触发ResponseNeverReceived时如何自定义爬虫关闭原因

方法1：全局捕获（下载器中间件实现） #

该方案可捕获整个爬虫项目所有请求抛出的ResponseNeverReceived异常，无需修改爬虫业务代码：

• 新建下载器中间件

from scrapy.exceptions import CloseSpider
from twisted.web._newclient import ResponseNeverReceived

class ExceptionCaptureMiddleware:
    def process_exception(self, request, exception, spider):
        # 匹配目标异常类型，触发带自定义原因的关闭
        if isinstance(exception, ResponseNeverReceived):
            close_reason = f"response_never_received|请求地址:{request.url}|异常详情:{str(exception)}"
            raise CloseSpider(reason=close_reason)

• 启用中间件
  在项目settings.py中注册中间件：

DOWNLOADER_MIDDLEWARES = {
    # 替换为你的中间件实际路径，优先级数值越小执行优先级越高
    'myproject.middlewares.ExceptionCaptureMiddleware': 90,
}

方法2：单请求捕获（errback回调实现） #

该方案适用于仅需要针对特定请求做异常监控的场景，无需修改全局配置：
在构造scrapy.Request时指定errback回调，在回调中判断异常类型：

import scrapy
from scrapy.exceptions import CloseSpider
from twisted.web._newclient import ResponseNeverReceived

class DemoSpider(scrapy.Spider):
    name = "demo"

    def start_requests(self):
        yield scrapy.Request(
            url="https://target-site.com",
            callback=self.parse,
            errback=self.request_errback
        )

    def parse(self, response):
        # 正常解析逻辑
        pass

    def request_errback(self, failure):
        # 校验异常类型
        if failure.check(ResponseNeverReceived):
            req = failure.request
            close_reason = f"response_never_received|请求地址:{req.url}|失败信息:{str(failure.value)}"
            raise CloseSpider(reason=close_reason)

生效验证 #

• 两种方案触发的自定义关闭原因，会直接作为spider_closed信号的reason参数返回
• 原有监听信号入库的逻辑无需调整，即可存储真实故障原因，方便后续问题定位

内容的提问来源于stack exchange，提问作者Sylvain