Scrapy爬虫突发Selector警告：传入text和root导致root被忽略求助

问题背景 #

原本正常运行的Scrapy爬虫突然弹出以下警告：

/home/user/github-repos/scrapper/scrapper/env/lib/python3.8/site-packages/scrapy/selector/unified.py:83: UserWarning: Selector got both text and root, root is being ignored. super().init(text=text, type=st, root=root, **kwargs)

排查发现是Scrapy内部Selector类的构造函数触发了该警告，相关代码如下：

__slots__ = ["response"]
selectorlist_cls = SelectorList

def __init__(self, response=None, text=None, type=None, root=None, **kwargs):
    if response is not None and text is not None:
        raise ValueError(
            f"{self.__class__.__name__}.__init__() received "
            "both response and text"
        )

    st = _st(response, type)

    if text is not None:
        response = _response_from_text(text, st)

    if response is not None:
        text = response.text
        kwargs.setdefault("base_url", response.url)

    self.response = response
    super().__init__(text=text, type=st, root=root, **kwargs)

警告提示root参数被忽略，但该构造函数又声明需要接收此参数，推测是Scrapy版本更新带来的兼容性问题。

本人代码中唯一与Selector交互的部分为ItemLoader的使用：

def load_item(self, response: TextResponse, app_id, db_id, urls):
    loader = AppLoader(response=response)
    loader.add_value("app_id", app_id)
    loader.add_value("db_id", db_id)
    loader.add_value("url", response.url)
    loader.add_css("game_title", "#appHubAppName::text")
    loader.add_css("publisher", "#game_highlights .dev_row+ .dev_row a::text")
    loader.add_css("developer", "#developers_list a::text")
    loader.add_css("publish_date", ".date::text")
    loader.add_css("tags", "#glanceCtnResponsiveRight a::text")
    loader.add_css(
        "review_count", "#review_type_all+ label .user_reviews_count::text"
    )
    loader.add_css(
        "positive_review_count",
        "#review_type_positive+ label .user_reviews_count::text",
    )
    loader.add_css(
        "negative_review_count",
        "#review_type_negative+ label .user_reviews_count::text",
    )
    loader.add_value("file_urls", urls)

    return loader.load_item()

解决方案 #

1. 锁定Scrapy稳定版本 #

若警告是在更新Scrapy后出现的，回退到之前能正常运行的版本。在requirements.txt中指定版本号：

scrapy==2.8.0

替换为你之前使用的稳定版本，然后重新安装依赖：

pip install -r requirements.txt --force-reinstall

2. 调整ItemLoader初始化方式 #

避免间接触发Selector同时接收text和root的情况，手动传入Selector对象而非直接传response：

from scrapy.selector import Selector

def load_item(self, response: TextResponse, app_id, db_id, urls):
    selector = Selector(response)
    loader = AppLoader(selector=selector)
    # 后续添加字段的代码保持不变

3. 临时屏蔽警告（不影响功能时使用） #

如果警告不影响爬虫正常运行，可在项目入口文件（如settings.py或爬虫主文件）中添加代码屏蔽该警告：

import warnings
warnings.filterwarnings("ignore", message="Selector got both text and root, root is being ignored.")

原因分析 #

Scrapy版本更新后，Selector父类的构造函数逻辑发生变更：当同时传入text和root参数时，会忽略root并抛出警告。而使用ItemLoader传入response时，内部会同时生成这两个参数传递给Selector，触发了新的参数检查逻辑。

内容的提问来源于stack exchange，提问作者limg21