Scrapy爬虫报错AttributeError: 'Spider'对象无'crawler'属性求助

问题背景 #

为在__init__中直接访问settings，自定义了from_crawler类方法，却导致Scrapy重试机制失效，触发AttributeError: 'Code1Spider' object has no attribute 'crawler'错误，使用Scrapy 2.0.1版本，运行在Zyte Cloud环境。

问题原因 #

Scrapy原生的from_crawler方法会自动将crawler对象绑定到爬虫实例的crawler属性上，而自定义的from_crawler仅返回传入settings的爬虫实例，未完成crawler属性绑定。当重试中间件尝试访问spider.crawler.stats统计重试数据时，就会触发属性不存在的错误。

同时，自定义的__init__方法仅接收settings参数，覆盖了Scrapy Spider基类默认的*args和**kwargs初始化逻辑，也会导致部分框架功能无法正常初始化。

解决方法 #

1. 调整__init__方法：保留基类的*args和**kwargs参数，调用基类初始化方法，确保框架原生逻辑正常执行。
2. 绑定crawler属性：在自定义的from_crawler方法中，创建爬虫实例后，手动将crawler对象赋值给爬虫的crawler属性。

修改后的爬虫代码 #

import scrapy
import datetime
import logging
# 假设load_gsheet和load_from_collection为已实现的自定义函数

class Code1Spider(scrapy.Spider):
    name = 'cointelegraph_pr'
    allowed_domains = ['cointelegraph.com']
    start_urls = ['https://cointelegraph.com/press-releases']
    
    def __init__(self, settings, *args, **kwargs):
        super().__init__(*args, **kwargs)  # 调用基类初始化方法，保留原生逻辑
        # 将settings转为字典格式
        settings = settings.copy_to_dict()
        
        self.id = int(str(datetime.now().timestamp()).split('.')[0])          
        self.gs_id = settings.get('GS_ID')
        self.endpoint_url = settings.get('ENDPOINT_URL')
        self.zyte_api_key = settings.get('ZYTE_API_KEY')        
        self.zyte_project_id = settings.get('ZYTE_PROJECT_ID')        
        self.zyte_collection_name = self.name
        
        # 加载谷歌表格中的停用词列表
        self.denied = load_gsheet(self.gs_id)
        # 加载Zyte集合中之前爬取过的URL
        self.scraped_urls = load_from_collection(self.zyte_project_id, self.zyte_collection_name, self.zyte_api_key)
        logging.info("###############################")
        logging.info("Number of previously scraped URLs = {}.".format(len(self.scraped_urls)))
        logging.info("")
        
    @classmethod
    def from_crawler(cls, crawler):
        settings = crawler.settings
        spider = cls(settings, crawler)  # 传入参数初始化爬虫实例
        spider.crawler = crawler  # 绑定crawler属性，供框架内部使用
        return spider

原始错误栈 #

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python3.8/site-packages/scrapy/core/downloader/middleware.py", line 42, in process_request
    defer.returnValue((yield download_func(request=request, spider=spider)))
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1362, in returnValue
    raise _DefGen_Return(val)
twisted.internet.defer._DefGen_Return: <504 https://cointelegraph.com/press-releases/the-launch-of-santa-browser-to-bring-in-the-next-200m-users-onto-web30>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python3.8/site-packages/scrapy/core/downloader/middleware.py", line 51, in process_response
    response = yield deferred_from_coro(method(request=request, response=response, spider=spider))
  File "/usr/local/lib/python3.8/site-packages/scrapy/downloadermiddlewares/retry.py", line 53, in process_response
    return self._retry(request, reason, spider) or response
  File "/usr/local/lib/python3.8/site-packages/scrapy/downloadermiddlewares/retry.py", line 69, in _retry
    stats = spider.crawler.stats
AttributeError: 'Code1Spider' object has no attribute 'crawler'

内容的提问来源于stack exchange，提问作者JBJ