Scrapy XPath报错：'Selector'对象无'_default_type'属性求助

问题重现 #

执行以下Scrapy Shell命令后，调用xpath查询时触发错误：

scrapy shell
fetch("https://www.worldometers.info/world-population/population-by-country/")
r = scrapy.Request(url="https://www.worldometers.info/world-population/population-by-country/")
fetch(r)
response.xpath('//h1/text()').get()

错误堆栈：

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-7a322073163e> in <module>
----> 1 response.xpath('//h1/text()').get()

~/anaconda3/envs/virtual_work/lib/python3.7/site-packages/scrapy/http/response/text.py in xpath(self, query, **kwargs)
    117 
    118     def xpath(self, query, **kwargs):
--> 119         return self.selector.xpath(query, **kwargs)
    120 
    121     def css(self, query):

~/anaconda3/envs/virtual_work/lib/python3.7/site-packages/scrapy/http/response/text.py in selector(self)
    113         from scrapy.selector import Selector
    114         if self._cached_selector is None:
--> 115             self._cached_selector = Selector(self)
    116         return self._cached_selector
    117 

~/anaconda3/envs/virtual_work/lib/python3.7/site-packages/scrapy/selector/unified.py in __init__(self, response, text, type, root, _root, **kwargs)
     84                             % self.__class__.__name__)
     85 
--> 86         st = _st(response, type or self._default_type)
     87 
     88         if _root is not None:
AttributeError: 'Selector' object has no attribute '_default_type'

原因分析 #

该错误多因旧版Scrapy的兼容性问题导致：重复调用fetch()并传入手动构造的Request对象时，会污染内部response对象的关联状态，使得Selector初始化时出现属性缺失异常。

解决方案 #

方案1：直接用URL调用fetch，跳过手动构造Request #

无需创建Request实例，直接通过URL重新获取响应：

scrapy shell
fetch("https://www.worldometers.info/world-population/population-by-country/")
# 需重新获取时，再次调用fetch(url)即可
fetch("https://www.worldometers.info/world-population/population-by-country/")
response.xpath('//h1/text()').get()

方案2：升级Scrapy至适配Python3.7的稳定版 #

你的环境使用Python3.7 + 旧版Scrapy，建议升级到兼容的稳定版本（如Scrapy 2.8.x系列）：

pip install --upgrade scrapy==2.8.0

升级后重启Scrapy Shell测试即可。

方案3：手动构造Selector实例绕过异常 #

若暂时无法升级，可直接基于响应文本手动创建Selector：

from scrapy.selector import Selector
sel = Selector(text=response.text)
sel.xpath('//h1/text()').get()

内容的提问来源于stack exchange，提问作者Xu阿兮