合并大体积Pickle字典时内存溢出，Jupyter内核崩溃求助

方法1：逐键写入新文件，避免全量加载 #

直接用原生pickle模块，先读取第一个字典，逐键值对写入新文件，写完后立即删除原字典释放内存，再处理第二个字典。这种方式内存中仅保留单个键值对和部分文件缓存，大幅降低内存占用。

import pickle

def merge_large_pickles(file1_path, file2_path, output_path):
    # 处理第一个Pickle文件
    with open(file1_path, 'rb') as src_f, open(output_path, 'wb') as dest_f:
        large_dict = pickle.load(src_f)
        for key, value in large_dict.items():
            pickle.dump((key, value), dest_f)
        del large_dict  # 强制释放内存
    
    # 处理第二个Pickle文件（追加写入）
    with open(file2_path, 'rb') as src_f, open(output_path, 'ab') as dest_f:
        large_dict = pickle.load(src_f)
        for key, value in large_dict.items():
            pickle.dump((key, value), dest_f)
        del large_dict

# 调用示例
merge_large_pickles('dict1.pkl', 'dict2.pkl', 'merged.pkl')

如果后续需要将键值对重新组装为字典，同样可以逐行读取合并后的文件，避免全量加载：

def load_merged_pickle(merged_path):
    merged_dict = {}
    with open(merged_path, 'rb') as f:
        while True:
            try:
                key, value = pickle.load(f)
                merged_dict[key] = value
            except EOFError:
                break
    return merged_dict

方法2：换用更高效的序列化格式 #

Pickle的内存占用较高，可先将两个字典转换为msgpack（一种轻量高效的序列化格式），再进行合并。msgpack序列化后的文件体积更小，加载时内存占用远低于Pickle。

import pickle
import msgpack

# 将Pickle转存为msgpack
def pickle_to_msgpack(pickle_path, msgpack_path):
    with open(pickle_path, 'rb') as f:
        data = pickle.load(f)
    with open(msgpack_path, 'wb') as f:
        msgpack.dump(data, f)
    del data

# 转换两个文件
pickle_to_msgpack('dict1.pkl', 'dict1.msgpack')
pickle_to_msgpack('dict2.pkl', 'dict2.msgpack')

# 合并msgpack文件
merged_dict = {}
with open('dict1.msgpack', 'rb') as f:
    merged_dict.update(msgpack.load(f))
with open('dict2.msgpack', 'rb') as f:
    merged_dict.update(msgpack.load(f))

# 保存合并结果
with open('merged.msgpack', 'wb') as f:
    msgpack.dump(merged_dict, f)

方法3：用数据库存储，避免全量内存加载 #

如果不需要一次性获取完整的合并字典，仅需按需查询，可将键值对存入SQLite数据库。这种方式完全不需要将所有数据加载到内存，从根源解决内存溢出问题。

import pickle
import sqlite3

# 初始化数据库
conn = sqlite3.connect('merged_dict.db')
cursor = conn.cursor()
cursor.execute('CREATE TABLE IF NOT EXISTS key_value_store (key TEXT PRIMARY KEY, value BLOB)')

# 写入第一个字典的数据
with open('dict1.pkl', 'rb') as f:
    data = pickle.load(f)
for key, value in data.items():
    cursor.execute('INSERT OR REPLACE INTO key_value_store VALUES (?, ?)', 
                  (key, pickle.dumps(value)))
conn.commit()
del data

# 写入第二个字典的数据
with open('dict2.pkl', 'rb') as f:
    data = pickle.load(f)
for key, value in data.items():
    cursor.execute('INSERT OR REPLACE INTO key_value_store VALUES (?, ?)', 
                  (key, pickle.dumps(value)))
conn.commit()
del data

# 按需查询示例
def get_value(key):
    cursor.execute('SELECT value FROM key_value_store WHERE key = ?', (key,))
    result = cursor.fetchone()
    return pickle.loads(result[0]) if result else None

# 使用示例
print(get_value('204747-44'))
conn.close()

额外注意事项 #

• 不要在Jupyter Lab中处理超大对象：Jupyter会保留变量引用，内存回收效率远低于纯Python脚本，建议将代码保存为.py文件直接运行。
• 运行前关闭其他占用内存的程序，尽可能释放系统可用内存。

内容的提问来源于stack exchange，提问作者Alice Green