如何进一步优化JSON响应片段转pd.DataFrame的加载效率？

我有一个包含大量大型JSON响应的pandas Series，之前尝试仅加载JSON对象的部分内容并转换为pd.DataFrame时，遇到内存占用过高、处理速度缓慢的问题。我写了下面的代码，已经规避了多数性能问题，但想知道有没有更高效的实现方式，寻求代码优化建议。

示例数据 #

{0: '{"city":"Campinas","bot-origin":null,"campaign-source":null,"lastState":"productAvailabilityCpfRequest","main-installation-date":"22/09/2021","userid":"c46528a7-988d-465e-9f12-25426a8b1808@tunnel.msging.net","full-name":"Claudenice lôbo da silva","alternative-installation-date":"23/09/2021","chosen-product":"Internet","bank":null,"postalcode":"13056015","due-date":"20","cpf":"30979696836","origin-link":null,"payment":"boleto","state":"SP","api-orders-hash-id":null,"email":"Silvaclaudenice71@gmail.com","plan-name":null,"userphone":"19 98715-0491","plan-offer":null,"completed-address":"13056015 - AV FERNANDO PAOLIERI, 182 - JARDIM PLANALTO DE VIRACOPOS, Campinas - SP","type-of-person":"CPF","type-of-product":"Residencial","main-installation-period-day":"manhã","plan-value":null,"alternative-installation-period-day":"manhã"}', 1: '{"city":"Campinas","bot-origin":null,"campaign-source":null,"lastState":"productAvailabilityStart","main-installation-date":"22/09/2021","userid":"c46528a7-988d-465e-9f12-25426a8b1808@tunnel.msging.net","full-name":"Claudenice lôbo da silva","alternative-installation-date":"23/09/2021","chosen-product":"Internet","bank":null,"postalcode":"13056015","due-date":"20","cpf":"30979696836","origin-link":null,"payment":"boleto","state":"SP","api-orders-hash-id":null,"email":"Silvaclaudenice71@gmail.com","plan-name":null,"userphone":"19 98715-0491","plan-offer":null,"completed-address":"13056015 - AV FERNANDO PAOLIERI, 182 - JARDIM PLANALTO DE VIRACOPOS, Campinas - SP","type-of-person":"CPF","type-of-product":"Residencial","main-installation-period-day":"manhã","plan-value":null,"alternative-installation-period-day":"manhã"}', 2: '{"city":"Campinas","bot-origin":null,"campaign-source":null,"lastState":"cpfValidationTrue","main-installation-date":"22/09/2021","userid":"c46528a7-988d-465e-9f12-25426a8b1808@tunnel.msging.net","full-name":"Claudenice lôbo da silva","alternative-installation-date":"23/09/2021","chosen-product":"Internet","bank":null,"postalcode":"13056015","due-date":"20","cpf":"30979696836","origin-link":null,"payment":"boleto","state":"SP","api-orders-hash-id":null,"email":"Silvaclaudenice71@gmail.com","plan-name":null,"userphone":"19 98715-0491","plan-offer":null,"completed-address":"13056015 - AV FERNANDO PAOLIERI, 182 - JARDIM PLANALTO DE VIRACOPOS, Campinas - SP","type-of-person":"CPF","type-of-product":"Residencial","main-installation-period-day":"manhã","plan-value":null,"alternative-installation-period-day":"manhã"}', 3: '{"city":"Campinas","bot-origin":null,"campaign-source":null,"lastState":"productAvailabilityCpfRequest","main-installation-date":"22/09/2021","userid":"c46528a7-988d-465e-9f12-25426a8b1808@tunnel.msging.net","full-name":"Claudenice lôbo da silva","alternative-installation-date":"23/09/2021","chosen-product":"Internet","bank":null,"postalcode":"13056015","due-date":"20","cpf":"30979696836","origin-link":null,"payment":"boleto","state":"SP","api-orders-hash-id":null,"email":"Silvaclaudenice71@gmail.com","plan-name":null,"userphone":"19 98715-0491","plan-offer":null,"completed-address":"13056015 - AV FERNANDO PAOLIERI, 182 - JARDIM PLANALTO DE VIRACOPOS, Campinas - SP","type-of-person":"CPF","type-of-product":"Residencial","main-installation-period-day":"manhã","plan-value":null,"alternative-installation-period-day":"manhã"}', 4: '{"city":"Campinas","bot-origin":null,"campaign-source":null,"lastState":"productAvailabilityStart","main-installation-date":"22/09/2021","userid":"c46528a7-988d-465e-9f12-25426a8b1808@tunnel.msging.net","full-name":"Claudenice lôbo da silva","alternative-installation-date":"23/09/2021","chosen-product":"Internet","bank":null,"postalcode":"13056015","due-date":"20","cpf":"30979696836","origin-link":null,"payment":"boleto","state":"SP","api-orders-hash-id":null,"email":"Silvaclaudenice71@gmail.com","plan-name":null,"userphone":"19 98715-0491","plan-offer":null,"completed-address":"13056015 - AV FERNANDO PAOLIERI, 182 - JARDIM PLANALTO DE VIRACOPOS, Campinas - SP","type-of-person":"CPF","type-of-product":"Residencial","main-installation-period-day":"manhã","plan-value":null,"alternative-installation-period-day":"manhã"}'}

现有代码 #

import orjson
def dataset_extras(extras, *args): # extras是传入的Series，args是需要提取的键列表
    l = [] 
    for i in extras:
        l.append({arg : orjson.loads(i).get(arg) for arg in args})
    return pd.DataFrame.from_records(l)

# 调用示例
dataset_extras(df.Extras,'city','campaign-source','api-orders-hash-id')

优化建议 #

1. 避免重复解析JSON #

当前代码每次循环都重复解析同一段JSON字符串，可先解析一次再提取所有需要的键，减少解析开销：

def dataset_extras(extras, *args):
    records = []
    for json_str in extras:
        data = orjson.loads(json_str)
        records.append({arg: data.get(arg) for arg in args})
    return pd.DataFrame.from_records(records)

2. 用pandas矢量化操作替代Python循环 #

利用pd.Series.apply结合自定义函数，借助pandas内部优化减少Python层面的循环开销：

def extract_keys(json_str, keys):
    data = orjson.loads(json_str)
    return {k: data.get(k) for k in keys}

def dataset_extras(extras, *args):
    extracted = extras.apply(lambda x: extract_keys(x, args))
    return pd.DataFrame(extracted.tolist())

3. 分块处理超大Series #

如果Series数据量极大导致内存紧张，可分块处理后拼接结果，降低单批次内存占用：

def dataset_extras(extras, *args, chunk_size=10000):
    chunks = []
    for start in range(0, len(extras), chunk_size):
        chunk = extras.iloc[start:start+chunk_size]
        records = []
        for json_str in chunk:
            data = orjson.loads(json_str)
            records.append({arg: data.get(arg) for arg in args})
        chunks.append(pd.DataFrame.from_records(records))
    return pd.concat(chunks, ignore_index=True)

4. 提前指定数据类型减少内存 #

创建DataFrame时，根据已知字段类型指定dtype，避免pandas自动推断带来的内存浪费：

def dataset_extras(extras, *args):
    records = []
    for json_str in extras:
        data = orjson.loads(json_str)
        records.append({arg: data.get(arg) for arg in args})
    # 根据实际字段类型调整映射
    dtype_map = {
        'city': 'string',
        'campaign-source': 'string',
        'api-orders-hash-id': 'string'
    }
    used_dtypes = {k: v for k, v in dtype_map.items() if k in args}
    return pd.DataFrame.from_records(records, dtype=used_dtypes)

5. 用orjson选项加速解析 #

orjson支持多种解析选项，若无需处理日期等特殊类型，可关闭相关校验进一步提升速度：

data = orjson.loads(json_str, option=orjson.OPT_DISABLE_STRICT_INTEGER_CHECK)

内容的提问来源于stack exchange，提问作者INGl0R1AM0R1