嵌套字典遍历报错：TypeError: bool类型不可迭代问题排查

尝试在大型嵌套字典中查找itemPrice键的所有出现位置时触发TypeError，提示“bool类型的参数不可迭代”。已确认顶级对象是dict类型，但字典体积过大无法手动排查内部是否存在非字典类型的值。

爬虫代码 #

import scrapy
    
class MlSpider(scrapy.Spider):
    name = 'detalhador'
    
    start_urls=['https://produto.mercadolivre.com.br/MLB-1304118411-sandalia-feminina-anabela-confortavel-pingente-mac-cod-133-_JM?attributes=COLOR_SECONDARY_COLOR%3AUHJldGE%3D%2CSIZE%3AMzU%3D&quantity=1']

    def parse(self, response,**kwargs):
        import json

        d = response.xpath('//script[contains(., "window.__PRELOADED_STATE__")]/text()').re_first(r'(?s)window.__PRELOADED_STATE__ = (.+?\});')
        data = json.loads(d)
        
        temp='itemPrice'
        res = [val[temp] for key, val in data.items() if temp in val]

报错堆栈信息 #

Traceback (most recent call last):
  File "/home/deborador/anaconda3/envs/mapsmaps/lib/python3.10/site-packages/scrapy/utils/defer.py", line 132, in iter_errback
    yield next(it)
  File "/home/deborador/anaconda3/envs/mapsmaps/lib/python3.10/site-packages/scrapy/utils/python.py", line 354, in __next__
    return next(self.data)
  File "/home/deborador/anaconda3/envs/mapsmaps/lib/python3.10/site-packages/scrapy/utils/python.py", line 354, in __next__
    return next(self.data)
  File "/home/deborador/anaconda3/envs/mapsmaps/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 66, in _evaluate_iterable
    for r in iterable:
  File "/home/deborador/anaconda3/envs/mapsmaps/lib/python3.10/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
    for x in result:
  File "/home/deborador/anaconda3/envs/mapsmaps/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 66, in _evaluate_iterable
    for r in iterable:
  File "/home/deborador/anaconda3/envs/mapsmaps/lib/python3.10/site-packages/scrapy/spidermiddlewares/referer.py", line 342, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/home/deborador/anaconda3/envs/mapsmaps/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 66, in _evaluate_iterable
    for r in iterable:
  File "/home/deborador/anaconda3/envs/mapsmaps/lib/python3.10/site-packages/scrapy/spidermiddlewares/urllength.py", line 40, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/deborador/anaconda3/envs/mapsmaps/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 66, in _evaluate_iterable
    for r in iterable:
  File "/home/deborador/anaconda3/envs/mapsmaps/lib/python3.10/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/deborador/anaconda3/envs/mapsmaps/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 66, in _evaluate_iterable
    for r in iterable:
  File "/home/deborador/Documentos/coding/mercadolivre/mercadolivre/mercadolivre/spiders/detalhador.py", line 22, in parse
    res = [val[temp] for key, val in data.items() if temp in val]
  File "/home/deborador/Documentos/coding/mercadolivre/mercadolivre/mercadolivre/spiders/detalhador.py", line 22, in <listcomp>
    res = [val[temp] for key, val in data.items() if temp in val]
TypeError: argument of type 'bool' is not iterable

部分字典片段 #

{"translations": {}, "initialState": {"id": "MLB1304118411", "variation_id": "46176185176", "layout": "vip-core", "vertical": "core", "components_locations": {"variations": "short_description"}, "components": {"head": [{"id": "compats_feedback", "type": "ui_message", "state": "HIDDEN", "closeable": false}, {"id": "related_searches", "type": "related_searches", "state": "VISIBLE", "title": {"text": "Você também pode gostar"}, "related_searches": [{"target": "https://lista.mercadolivre.com.br/chinelo-comfortflex#topkeyword", "timeout": 0, "duration": 0, "label": {"text": "chinelo comfortflex"}}, {"target": "https://lista.mercadolivre.com.br/calçados-andacco#topkeyword", "timeout": 0, "duration": 0, "label": {"text": "calcados andacco"}}, {"target": "https://lista.mercadolivre.com.br/chinelo-usaflex#topkeyword", "timeout": 0, "duration": 0, "label": {"text": "chinelo usaflex"}}, {"target": "https://lista.mercadolivre.com.br/chinelo-ortopedico#topkeyword", "timeout": 0, "duration": 0, "label": {"text": "chinelo ortopedico"}}, {"target": "https://lista.mercadolivre.com.br/chinelo-crocs-feminino#topkeyword", "timeout": 0, "duration": 0, "label": {"text": "chinelo crocs feminino"}}, {"target": "https://lista.mercadolivre.com.br/ramarim-sandalia#topkeyword", "timeout": 0, "duration": 0, "label": {"text": "ramarim sandalia"}}, {"target": "https://lista.mercadolivre.com.br/melissa-oficial#topkeyword", "timeout": 0, "duration": 0, "label": {"text": "melissa oficial"}}]}, {"id": "carousel_cheaper", "type": "carousel", "state": "VISIBLE", "carousel": {}, "carousel_config": {"site_id": "MLB", "item_id": "MLB1304118411", "category_id": "MLB273770", "client": "similar"}}}

解决方法 #

问题根源 #

列表推导式中if temp in val的判断逻辑有两个问题：

1. 当val是bool类型（如False）时，in操作会尝试迭代该bool值，直接触发TypeError；
2. 仅遍历顶级字典的value，无法处理嵌套的子字典和列表，根本找不到深层的itemPrice。

1. 临时修复（仅遍历顶级字典） #

先判断val是否为dict类型，再检查键是否存在：

res = [val[temp] for key, val in data.items() if isinstance(val, dict) and temp in val]

2. 完整解决方案（递归遍历所有嵌套层级） #

编写递归函数，遍历所有嵌套的字典和列表，收集所有itemPrice的值：

def find_item_prices(obj, key='itemPrice'):
    results = []
    if isinstance(obj, dict):
        for k, v in obj.items():
            if k == key:
                results.append(v)
            # 递归遍历子结构
            results.extend(find_item_prices(v, key))
    elif isinstance(obj, list):
        for item in obj:
            results.extend(find_item_prices(item, key))
    # 非字典/列表类型直接跳过
    return results

# 在parse函数中调用
temp = 'itemPrice'
res = find_item_prices(data, temp)

这个函数会自动处理所有嵌套结构，遇到bool、字符串、数字等基础类型时直接跳过，既不会触发迭代错误，又能找到所有层级的itemPrice值。

内容的提问来源于stack exchange，提问作者user1563977