Scrapy爬取API时报TypeError: 'NoneType' object is not iterable如何解决

错误原因 #

• 回调函数复用逻辑错误：你用同一个parse方法处理两种完全不同结构的接口返回：
  1. 第一个列表接口https://api.pointsbet.com/api/v2/competitions/7176/events/featured返回值包含events字段
  2. 后续的赛事详情接口返回值没有events字段，调用resp.get('events')会返回None，循环None就会抛出你遇到的类型错误
• 异步请求逻辑错误：Scrapy的yield scrapy.Request是异步操作，代码执行到yield后不会等待请求返回，会直接向后执行，此时你拿到的response仍然是列表接口的响应，列表接口没有fixedOddsMarkets字段，获取到的值也是None，循环时也会触发同类型错误
• 缺少空值校验：没有对接口返回的字段做非空判断，只要接口返回结构变动或者字段缺失就会直接报错

修正后代码 #

import scrapy
import json


class PbSpider(scrapy.Spider):
    name = 'pb'
    allowed_domains = ['api.pointsbet.com']
    start_urls = ['https://api.pointsbet.com/api/v2/competitions/7176/events/featured']

    def parse(self, response):
        # 处理赛事列表接口返回
        resp = json.loads(response.body)
        events = resp.get('events')
        # 非空校验避免报错
        if not events:
            return
        for event in events:
            eventKey = event.get('key')
            if not eventKey:
                continue
            gameapi = f'https://api.pointsbet.com/api/v2/events/{eventKey}'
            # 赛事详情请求交给单独的回调方法处理
            yield scrapy.Request(url=gameapi, callback=self.parse_game)
    
    def parse_game(self, response):
        # 处理单个赛事详情接口返回
        res = json.loads(response.body)
        fixedOddsMarkets = res.get('fixedOddsMarkets')
        if not fixedOddsMarkets:
            return
        for fixedOddsMarket in fixedOddsMarkets:
            market = fixedOddsMarket.get('eventName')
            outcomes = fixedOddsMarket.get('outcomes')
            if not outcomes:
                continue
            for outcome in outcomes:
                name = outcome.get('name')
                price = outcome.get('price')
                print(market, name, price)

内容的提问来源于stack exchange，提问作者Geordie Frost