如何更新Pickle文件？误覆盖内容后的正确更新方法

你的操作错误原因 #

你用了'wb'模式打开文件——这个模式是写入二进制模式，会直接清空文件原有内容，所以当你只dump了X1_test和X2_test后，原来的所有数据都被覆盖了，自然文件大小大幅缩小。

Pickle文件本质是序列化的数据流，它不像数据库支持随机修改某一部分内容，所以直接追加或覆盖某段数据是不可行的，必须遵循“全读-修改-全写”的流程。

解决方法1：按原dump顺序读取并重新写入 #

因为你当初是逐个变量dump到文件里的，所以读取时必须严格按照原dump的顺序把所有变量都加载出来，替换需要更新的变量后，再重新全部写入文件：

import pickle

# 第一步：读取原文件的所有变量，顺序必须和当初dump的完全一致
with open('variables.pkl', 'rb') as f:
    embedding_weights = pickle.load(f)
    X1 = pickle.load(f)
    X2 = pickle.load(f)
    Y = pickle.load(f)
    # 跳过旧的X1_test和X2_test，直接用新值替换
    _ = pickle.load(f)
    _ = pickle.load(f)
    Y_test = pickle.load(f)
    X1_nli = pickle.load(f)
    X2_nli = pickle.load(f)
    Y_nli = pickle.load(f)
    X1_test_nli = pickle.load(f)
    X2_test_nli = pickle.load(f)
    Y_test_nli = pickle.load(f)
    X1_test_matched = pickle.load(f)
    X2_test_matched = pickle.load(f)
    Y_test_matched = pickle.load(f)
    X1_test_mismatched = pickle.load(f)
    X2_test_mismatched = pickle.load(f)
    Y_test_mismatched = pickle.load(f)
    X2_two_sentences = pickle.load(f)
    X2_test_two_sentences = pickle.load(f)
    tokenizer = pickle.load(f)

# 第二步：替换为更新后的X1_test和X2_test
# 假设你已经有了更新后的X1_test和X2_test变量

# 第三步：重新写入所有变量到文件
with open('variables.pkl', 'wb') as output:
    pickle.dump(embedding_weights, output, 2)
    pickle.dump(X1, output, 2)
    pickle.dump(X2, output, 2)
    pickle.dump(Y, output, 2)
    pickle.dump(X1_test, output, 2)  # 写入更新后的变量
    pickle.dump(X2_test, output, 2)  # 写入更新后的变量
    pickle.dump(Y_test, output, 2)
    pickle.dump(X1_nli, output, 2)
    pickle.dump(X2_nli, output, 2)
    pickle.dump(Y_nli, output, 2)
    pickle.dump(X1_test_nli, output, 2)
    pickle.dump(X2_test_nli, output, 2)
    pickle.dump(Y_test_nli, output, 2)
    pickle.dump(X1_test_matched, output, 2)
    pickle.dump(X2_test_matched, output, 2)
    pickle.dump(Y_test_matched, output, 2)
    pickle.dump(X1_test_mismatched, output, 2)
    pickle.dump(X2_test_mismatched, output, 2)
    pickle.dump(Y_test_mismatched, output, 2)
    pickle.dump(X2_two_sentences, output, 2)
    pickle.dump(X2_test_two_sentences, output, 2)
    pickle.dump(tokenizer, output, 2)

解决方法2：用字典封装变量（更推荐，后续维护更简单） #

如果后续还要修改变量，建议把所有数据封装到一个字典里保存，这样不用记住变量的dump顺序，操作更灵活：

（如果是第一次保存）

data = {
    'embedding_weights': embedding_weights,
    'X1': X1,
    'X2': X2,
    'Y': Y,
    'X1_test': X1_test,
    'X2_test': X2_test,
    'Y_test': Y_test,
    'X1_nli': X1_nli,
    'X2_nli': X2_nli,
    'Y_nli': Y_nli,
    'X1_test_nli': X1_test_nli,
    'X2_test_nli': X2_test_nli,
    'Y_test_nli': Y_test_nli,
    'X1_test_matched': X1_test_matched,
    'X2_test_matched': X2_test_matched,
    'Y_test_matched': Y_test_matched,
    'X1_test_mismatched': X1_test_mismatched,
    'X2_test_mismatched': X2_test_mismatched,
    'Y_test_mismatched': Y_test_mismatched,
    'X2_two_sentences': X2_two_sentences,
    'X2_test_two_sentences': X2_test_two_sentences,
    'tokenizer': tokenizer
}

with open('variables.pkl', 'wb') as output:
    pickle.dump(data, output, 2)

（更新变量时）

import pickle

# 读取整个字典
with open('variables.pkl', 'rb') as f:
    data = pickle.load(f)

# 直接修改指定键对应的值
data['X1_test'] = updated_X1_test
data['X2_test'] = updated_X2_test

# 重新写入整个字典
with open('variables.pkl', 'wb') as f:
    pickle.dump(data, f, 2)

这种方式不需要记住变量的顺序，后续添加或修改变量都更方便，也不容易出错。

内容的提问来源于stack exchange，提问作者Shawn