Python中json.loads截断API返回数据的问题排查与解决

我在解析API返回的JSON数据时遇到了问题，以下是我用来获取所有分页数据的代码：

try:
    limit = 100         # 每页最大条数，可按需调整
    page = 1            # 从第一页开始
    headers = {
        'Authorization': 'Token {}'.format(api_token),
        'Content-Type': 'application/json'}

    results = []
    while True:
        headers['X-Page-Limit'] = str(limit)
        headers['X-Page-Page'] = str(page)

        response = requests.get(url, headers=headers)
        counter += 1
        
        if 200 <= response.status_code < 300:
            results.extend(response.json())
            response_headers = response.headers
            limit = int(response_headers.get('X-Page-Limit', limit))
            page = int(response_headers.get('X-Page-Page', page)) + 1
            total = int(response_headers.get('X-Page-Total', 0))

            if page > (total // limit) + (total % limit > 0):       ## 检查是否已获取所有页面
                print ('Before loads: ',len(results))
                print ('Before loads: ',results)
                results = json.loads(response.content)
                print ('After loads: ',len(results))
                print ('After loads: ',results)

                return results, counter
                break

调用json.loads()前，print(results)显示有169条记录，但调用后只剩69条。部分记录的description字段包含引号或单引号，不确定是否因此导致解析异常。

同时，results在调用前后内容差异明显：

调用前仅显示无值的键列表：
['auto_upgrade', 'rtsa', 'wifi', 'switch_mgmt', 'wootcloud', 'rogue', 'mgmt', 'persist_config_on_device', 'uplink_port_config', 'engagement', 'analytic', 'ssh_keys', 'vars', 'wids', 'mxtunnel', 'occupancy', 'public_zone_occupancy', 'zone_occupancy_alert', 'gateway_mgmt', 'ssr', 'mxedge', 'synthetic_test', 'led', 'status_portal', 'config_auto_revert', 'enable_unii_4', 'id', 'for_site', 'site_id', 'org_id', 'created_time', 'modified_time']

调用后显示完整数据：
{'auto_upgrade': {'enabled': True, 'version': 'custom', 'time_of_day': '23:00', 'custom_versions': {'AP12': '0.12.27139', 'AP21': '', 'AP24': '0.12.27139', 'AP32': '0.12.27139', 'AP32E': '0.12.27139', 'AP33': '0.12.27139', 'AP34': '0.12.27139', 'AP41': '', 'AP41E': '', 'AP43': '', 'AP43E': '', 'AP45': '0.12.27139', 'AP45E': '0.12.27139', 'AP61': '', 'AP61E': '', 'AP63': '', 'AP63E': '', 'AP64': '0.12.27139'}, 'day_of_week': 'thu'}, 'rtsa': {'enabled': True, 'track_asset': True, 'app_waking': True}, 'wifi': {'enabled': True, 'locate_unconnected': True, 'mesh_enabled': False, 'mesh_allow_dfs': False, 'detect_interference': False, 'locate_connected': False}, 'switch_mgmt': {'use_mxedge_proxy': False, 'root_password': 'some'}, 'wootcloud': None, 'rogue': {'min_rssi': -85, 'min_duration': 10, 'enabled': True, 'honeypot_enabled': True, 'whitelisted_bssids': [], 'whitelisted_ssids': []}, 'mgmt': {'use_wxtunnel': False}, 'persist_config_on_device': True, 'uplink_port_config': {'keep_wlans_up_if_down': False}, 'engagement': {'dwell_tags': {'passerby': '1-300', 'bounce': '301-14400', 'engaged': '14401-28800', 'stationed': '28801-42000'}, 'dwell_tag_names': {'passerby': '', 'bounce': '', 'engaged': '', 'stationed': ''}, 'hours': {'sun': None, 'mon': None, 'tue': None, 'wed': None, 'thu': None, 'fri': None, 'sat': None}}, 'analytic': {'enabled': True}, 'ssh_keys': [], 'vars': {}, 'wids': {}, 'mxtunnel': {'enabled': False}, 'occupancy': {'min_duration': 120, 'clients_enabled': True, 'sdkclients_enabled': True, 'assets_enabled': True, 'unconnected_clients_enabled': True}, 'public_zone_occupancy': {'enabled': True, 'client_density_enabled': False, 'rssi_zones_enabled': False, 'url': 'https://app.eu.mist.com/something'}, 'zone_occupancy_alert': {'enabled': False, 'threshold': 5}, 'gateway_mgmt': {'app_usage': False, 'security_log_source_interface': '', 'auto_signature_update': {'enable': True, 'time_of_day': '02:00', 'day_of_week': ''}}, 'ssr': {}, 'mxedge': {'radsec': {'enabled': False, 'server_selection': 'ordered', 'match_ssid': False, 'auth_servers': [], 'acct_servers': [], 'src_ip_source': 'any'}, 'mist_nac': {'enabled': False, 'client_ips': {}, 'auth_server_port': 1812, 'acct_server_port': 1812}, 'mist_nacedge': {'enabled': False, 'auth_ttl': 604800, 'mxedge_hosts': []}}, 'synthetic_test': {'-wan_speedtest': ''}, 'led': {'enabled': True, 'brightness': 255}, 'status_portal': {'enabled': False, 'hostnames': []}, 'config_auto_revert': True, 'enable_unii_4': False, 'id': 'portalid', 'for_site': True, 'site_id': 'siteid', 'org_id': 'orgid', 'created_time': 0, 'modified_time': 10}

不过，另一种分页获取API数据的方法能正确返回全部169条记录：

try:
    results = []
    page = 1
    limit = 100
    params = {'X-Page-Page': page, 'X-Page-Total': 109}

    pages_remaining = True
    while pages_remaining:
        response = requests.get(check_devices_stats_url, headers=headers, params=params).json()
        if response:
            [results.append(i) for i in (response)]
            page += 1
            params = {'X-Page-Page' : page}
            if not len(response) == 100:
                pages_remaining = False

更新 #

针对REST API可能返回列表或字典的复用函数问题，我采用了以下处理方式：

try:
    limit = 100
    page = 1
    headers = {
        'Authorization': 'Token {}'.format(api_token),
        'Content-Type': 'application/json'}

    results = []
    while True:
        headers['X-Page-Limit'] = str(limit)
        headers['X-Page-Page'] = str(page)

        response = requests.get(url, headers=headers)
        counter += 1
        
        if 200 <= response.status_code < 300:
            if type (response.json()) is list:
                results = results + response.json()         # 响应为列表时，将新数据追加到自定义列表
            elif isinstance(response.json(), dict):         # 使用type() is dict无法匹配字典子类，改用isinstance
                results.append(response.json())             # 响应为字典（单个对象）时，将其追加到自定义列表

            response_headers = response.headers
            limit = int(response_headers.get('X-Page-Limit', limit))
            page = int(response_headers.get('X-Page-Page', page)) + 1
            total = int(response_headers.get('X-Page-Total', 0))

            if page > (total // limit) + (total % limit > 0):
                return results, counter
                break

内容的提问来源于stack exchange，提问作者ZeusTM