Scrapy运行报错：cannot pickle 'module' object 问题求助

问题场景 #

按照RealPython的《使用Scrapy和MongoDB进行网络爬取》教程操作时，运行Scrapy代码触发以下错误：

File "/home/nawalnaz/miniconda3/bin/scrapy", line 8, in <module>
    sys.exit(execute())
             ^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/site-packages/scrapy/cmdline.py", line 160, in execute
    cmd.crawler_process = CrawlerProcess(settings)
                          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/site-packages/scrapy/crawler.py", line 357, in __init__
    super().__init__(settings)
  File "/home/nawalnaz/miniconda3/lib/python3.11/site-packages/scrapy/crawler.py", line 227, in __init__
    self.spider_loader = self._get_spider_loader(settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/site-packages/scrapy/crawler.py", line 221, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
                                    ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/site-packages/scrapy/settings/__init__.py", line 470, in frozencopy
    copy = self.copy()
           ^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/site-packages/scrapy/settings/__init__.py", line 452, in copy
    return copy.deepcopy(self)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 161, in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle 'module' object

问题代码定位 #

经排查，错误源于MongoDBPipeline类的初始化代码：

class MongoDBPipeline(object):
    def __init__(self):
        connection = pymongo.MongoClient(
            settings['MONGODB_SERVER'],
            settings['MONGODB_PORT']
        )
        db = connection[settings['MONGODB_DB']]
        self.collection = db[settings['MONGODB_COLLECTION']]

修复方案 #

问题核心是直接引用了全局的settings模块对象，而Scrapy在处理设置时会尝试深拷贝操作，模块对象无法被序列化/深拷贝。改用Scrapy推荐的from_crawler类方法获取设置即可解决：

import pymongo
from scrapy.utils.project import get_project_settings

class MongoDBPipeline(object):
    @classmethod
    def from_crawler(cls, crawler):
        # 通过crawler实例安全获取项目设置
        settings = crawler.settings
        pipeline = cls()
        # 初始化MongoDB连接与集合
        connection = pymongo.MongoClient(
            settings.get('MONGODB_SERVER'),
            settings.get('MONGODB_PORT')
        )
        db = connection[settings.get('MONGODB_DB')]
        pipeline.collection = db[settings.get('MONGODB_COLLECTION')]
        return pipeline

    def __init__(self):
        # 保留空初始化方法，避免实例化报错
        self.collection = None

关键说明 #

• from_crawler是Scrapy官方推荐的Pipeline初始化方式，能避免直接引用全局模块导致的序列化问题；
• 使用settings.get()替代直接索引settings[]，可避免因配置键不存在引发的KeyError，代码更健壮。

内容的提问来源于stack exchange，提问作者kahanahmed3003