Python调用BallDontLie API批量请求崩溃，遇JSONDecodeError求助

使用Python的requests库调用BallDontLie的season_averages API获取球员赛季数据时，单赛季小范围请求可正常返回结果，但遍历1990-2021所有赛季+多个player_id的嵌套循环中，代码会随机崩溃，抛出JSONDecodeError: Expecting value: line 1 column 1 (char 0)错误，且每次崩溃位置不固定。怀疑是API限流导致，但不想拆分生成多个DataFrame再合并，求可行解决方案。

原代码 #

players_stats=[]

seasons=[1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,
       2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,
       2012,2013,2014,2015,2016,2017,2018,2019,2020,2021]

players_game_stats_url="https://www.balldontlie.io/api/v1/season_averages?"
print("Retrieving Player Ids")
print("-"*20)

for season in seasons:
    for player_id in players_ids:
        try:
            player_game_stats=requests.get(players_game_stats_url, params={"season":season,"player_ids[]":player_id}).json()
            player_average_pts=player_game_stats['data'][0]['pts']
            player_avg_assists=player_game_stats['data'][0]['ast']
            player_avg_blocks=player_game_stats['data'][0]['blk']
            player_avg_defrebounds=player_game_stats['data'][0]['dreb']
            player_avg_offrebounds=player_game_stats['data'][0]['reb']
            player_avg_3pt_attempts=player_game_stats['data'][0]['fg3a']
            player_avg_3pt_made=player_game_stats['data'][0]['fg3m']
            player_avg_2pt_attempts=player_game_stats['data'][0]['fga']
            player_avg_2pt_made=player_game_stats['data'][0]['fgm']
            player_avg_ft_attempts=player_game_stats['data'][0]['fta']
            player_avg_ft_made=player_game_stats['data'][0]['ftm']
            player_avg_steals=player_game_stats['data'][0]['stl']
            player_avg_mins=player_game_stats['data'][0]['min']
            player_avg_turnovers=player_game_stats['data'][0]['turnover']
            
            players_stats.append({"Player ID": player_id,"Season":season,"Average Points": player_average_pts,
                                  "Average Assists": player_avg_assists, "Average Blocks":player_avg_blocks, 
                                  "Average Defensive Rebounds":player_avg_defrebounds, 
                                  "Average Offensive Rebounds":player_avg_offrebounds, 
                                  "Average 3PT Attempts":player_avg_3pt_attempts,
                                 "Average 3PT Made":player_avg_3pt_made, "Average 2PT Attempts":player_avg_2pt_made,
                                  "Average 2PT Made":player_avg_2pt_made,
                                 "Average Free Throw Attempts":player_avg_ft_attempts, 
                                  "Average Free Throws Made":player_avg_ft_made, "Average Steals per Game":player_avg_steals,
                                 "Average Mins per Game":player_avg_mins,"Average Turn Overs per Game":player_avg_turnovers })

        except (IndexError):
            print(f"The stats for player:{player_id} were not found for season:{season}")
            pass
        
players_stats

单赛季（1990年）运行结果 #

Retrieving Player Ids
--------------------
The stats for player:671 were not found for 1990
The stats for player:735 were not found for 1990
The stats for player:847 were not found for 1990
The stats for player:907 were not found for 1990
The stats for player:957 were not found for 1990
The stats for player:1037 were not found for 1990
The stats for player:1089 were not found for 1990
The stats for player:1179 were not found for 1990
The stats for player:1247 were not found for 1990
The stats for player:1281 were not found for 1990
The stats for player:1368 were not found for 1990
The stats for player:1445 were not found for 1990
The stats for player:237 were not found for 1990
The stats for player:220 were not found for 1990
The stats for player:1593 were not found for 1990
[{'Player ID': 2895,
  'Average Points': 18.43,
  'Average Assists': 2.2,
  'Average Blocks': 1.34,
  'Average Defensive Rebounds': 6.62,
  'Average Offensive Rebounds': 10.26,
  'Average 3PT Attempts': 0.51,
  'Average 3PT Made': 0.18,
  'Average 2PT Attempts': 6.95,
  'Average 2PT Made': 6.95,
  'Average Free Throw Attempts': 5.97,
  'Average Free Throws Made': 4.36,
  'Average Steals per Game': 0.96,
  'Average Mins per Game': '35:09',
  'Average Turn Overs per Game': 2.93}]

批量运行报错信息 #

Retrieving Player Ids
--------------------
The stats for player:671 were not found for season:1990
The stats for player:735 were not found for season:1990
The stats for player:847 were not found for season:1990
---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
File ~\anaconda3\lib\site-packages\requests\models.py:910, in Response.json(self, **kwargs)
    909 try:
--> 910     return complexjson.loads(self.text, **kwargs)
    911 except JSONDecodeError as e:
    912     # Catch JSON-related errors and raise as requests.JSONDecodeError
    913     # This aliases json.JSONDecodeError and simplejson.JSONDecodeError

File ~\anaconda3\lib\json\__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    343 if (cls is None and object_hook is None and
    344         parse_int is None and parse_float is None and
    345         parse_constant is None and object_pairs_hook is None and not kw):
--> 346     return _default_decoder.decode(s)
    347 if cls is None:

File ~\anaconda3\lib\json\decoder.py:337, in JSONDecoder.decode(self, s, _w)
    333 """Return the Python representation of ``s`` (a ``str`` instance
    334 containing a JSON document).
    335 
    336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338 end = _w(s, end).end()

File ~\anaconda3\lib\json\decoder.py:355, in JSONDecoder.raw_decode(self, s, idx)
    354 except StopIteration as err:
--> 355     raise JSONDecodeError("Expecting value", s, err.value) from None
    356 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

JSONDecodeError                           Traceback (most recent call last)
Input In [8], in <cell line: 11>()
     12 for player_id in players_ids:
     13     try:
---> 14         player_game_stats=requests.get(players_game_stats_url, params={"season":season,"player_ids[]":player_id}).json()
     15         player_average_pts=player_game_stats['data'][0]['pts']
     16         player_avg_assists=player_game_stats['data'][0]['ast']

File ~\anaconda3\lib\site-packages\requests\models.py:917, in Response.json(self, **kwargs)
    915     raise RequestsJSONDecodeError(e.message)
    916 else:
--> 917     raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)

JSONDecodeError: [Errno Expecting value] Retry later
: 0

1. 增加请求间隔，降低触发限流概率 #

在循环中加入time.sleep()控制请求频率，避免短时间内发送过多请求：

import time

# 在每次请求后添加1秒间隔
for season in seasons:
    for player_id in players_ids:
        try:
            player_game_stats=requests.get(players_game_stats_url, params={"season":season,"player_ids[]":player_id}).json()
            # ... 数据提取和存储逻辑
            time.sleep(1)  # 正常请求后暂停1秒
        except:
            # ... 异常处理逻辑

2. 捕获解码错误并自动重试 #

遇到JSONDecodeError时，自动重试请求，避免程序直接崩溃，同时针对限流状态码（429）延长等待时间：

import time
from requests.exceptions import JSONDecodeError

max_retries = 3
for season in seasons:
    for player_id in players_ids:
        retries = 0
        success = False
        while retries < max_retries and not success:
            try:
                response = requests.get(players_game_stats_url, params={"season":season,"player_ids[]":player_id})
                # 检查响应状态码
                if response.status_code == 429:
                    print(f"触发限流，玩家{player_id}赛季{season}将重试")
                    time.sleep(5)
                    retries +=1
                    continue
                elif response.status_code != 200:
                    print(f"请求失败，状态码{response.status_code}，玩家{player_id}赛季{season}")
                    retries +=1
                    time.sleep(2)
                    continue
                
                player_game_stats = response.json()
                # ... 数据提取和存储逻辑
                success = True
                time.sleep(1)
            except IndexError:
                print(f"玩家{player_id}赛季{season}无数据")
                success = True
            except JSONDecodeError:
                print(f"解码失败，玩家{player_id}赛季{season}重试中...")
                retries +=1
                time.sleep(3)

3. 批量请求减少请求次数 #

BallDontLie API支持一次性传入多个player_id，将玩家ID分组批量请求，大幅降低总请求数：

from itertools import islice
import time

# 定义分组函数，每10个玩家一组
def chunk(it, size):
    it = iter(it)
    return iter(lambda: tuple(islice(it, size)), ())

for season in seasons:
    # 按10个玩家为一组批量请求
    for player_chunk in chunk(players_ids, 10):
        params = {"season": season, "player_ids[]": player_chunk}
        response = requests.get(players_game_stats_url, params=params)
        if response.status_code == 200:
            data = response.json()['data']
            # 遍历返回的每个玩家数据
            for item in data:
                players_stats.append({
                    "Player ID": item['player_id'],
                    "Season": season,
                    "Average Points": item['pts'],
                    "Average Assists": item['ast'],
                    # ... 其他字段对应提取
                })
        time.sleep(1)

4. 完善异常捕获范围 #

将请求相关的所有异常（如连接超时、HTTP错误）都纳入捕获，避免程序意外终止：

from requests.exceptions import RequestException, JSONDecodeError

try:
    response = requests.get(...)
    response.raise_for_status()  # 触发HTTP错误抛出异常
    player_game_stats = response.json()
    # ... 数据处理逻辑
except IndexError:
    # 处理无数据情况
except (RequestException, JSONDecodeError):
    # 处理请求失败或解码错误

内容的提问来源于stack exchange，提问作者Youssouf