在Python中为CNN模型加载gz文件时遇反序列化错误的解决方法

问题原因 #

你直接用pickle.load()读取了.tar.gz压缩归档文件，该文件并非pickle序列化格式，内部包含多个独立的pickle数据集文件，因此触发解包错误。

解决方案 #

方案1：手动解压后读取pickle文件 #

• 用压缩工具（如7-Zip、WinRAR）解压cifar-10-python.tar.gz，得到cifar-10-batches-py文件夹
• 文件夹内的data_batch_1至data_batch_5是训练集文件，test_batch是测试集文件，这些才是可被pickle读取的文件
• 修改代码加载数据：

import pickle
import numpy as np

def unpickle(file):
    with open(file, 'rb') as fo:
        dict = pickle.load(fo, encoding='bytes')
    return dict

# 加载训练集
training_data = []
training_labels = []
for i in range(1, 6):
    batch = unpickle(f'cifar-10-batches-py/data_batch_{i}')
    training_data.append(batch[b'data'])
    training_labels.append(batch[b'labels'])
training_images = np.concatenate(training_data).reshape(-1, 3, 32, 32).transpose(0, 2, 3, 1)
training_labels = np.concatenate(training_labels)

# 加载测试集
test_batch = unpickle('cifar-10-batches-py/test_batch')
testing_images = test_batch[b'data'].reshape(-1, 3, 32, 32).transpose(0, 2, 3, 1)
testing_labels = np.array(test_batch[b'labels'])

方案2：代码自动解压读取 #

借助tarfile模块在代码内直接解压并读取数据：

import pickle
import tarfile
import numpy as np

def load_cifar10_from_tar(tar_path):
    training_data = []
    training_labels = []
    testing_data = []
    testing_labels = []
    
    with tarfile.open(tar_path, 'r:gz') as tar:
        # 遍历读取训练集批次
        for i in range(1, 6):
            batch_file = tar.extractfile(f'cifar-10-batches-py/data_batch_{i}')
            batch = pickle.load(batch_file, encoding='bytes')
            training_data.append(batch[b'data'])
            training_labels.append(batch[b'labels'])
        
        # 读取测试集
        test_file = tar.extractfile('cifar-10-batches-py/test_batch')
        test_batch = pickle.load(test_file, encoding='bytes')
        testing_data.append(test_batch[b'data'])
        testing_labels.append(test_batch[b'labels'])
    
    # 转换为标准图像格式（样本数, 高, 宽, 通道数）
    training_images = np.concatenate(training_data).reshape(-1, 3, 32, 32).transpose(0, 2, 3, 1)
    training_labels = np.concatenate(training_labels)
    testing_images = np.concatenate(testing_data).reshape(-1, 3, 32, 32).transpose(0, 2, 3, 1)
    testing_labels = np.concatenate(testing_labels)
    
    return (training_images, training_labels), (testing_images, testing_labels)

# 调用函数加载数据
local_dataset_path = 'cifar-10-python.tar.gz'
(training_images, training_labels), (testing_images, testing_labels) = load_cifar10_from_tar(local_dataset_path)

方案3：使用Keras内置数据集（最简便） #

你已导入datasets模块，可直接调用官方API加载，无需手动处理压缩包：

from keras._tf_keras.keras.datasets import cifar10

(training_images, training_labels), (testing_images, testing_labels) = cifar10.load_data()

内容的提问来源于stack exchange，提问作者Luis Cruz