在Google Colab加载.npy文件遇UnpicklingError问题求助

本地Jupyter Notebook（Python 3.9.13 + NumPy 1.24.3）能正常加载的ground_truth.npy文件，在Google Colab（Python 3.10.12 + NumPy 1.23.5）中加载时出现以下错误：

报错信息 #

当设置allow_pickle=True时：

UnpicklingError                           Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding, max_header_size)
    440             try:
--> 441                 return pickle.load(fid, **pickle_kwargs)
    442             except Exception as e:

UnpicklingError: invalid load key, '\xef'.

The above exception was the direct cause of the following exception:

UnpicklingError                           Traceback (most recent call last)
1 frames
/usr/local/lib/python3.10/dist-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding, max_header_size)
    441                 return pickle.load(fid, **pickle_kwargs)
    442             except Exception as e:
--> 443                 raise pickle.UnpicklingError(
    444                     print("Hello world")
    445                     f"Failed to interpret file {file!r} as a pickle") from e

UnpicklingError: Failed to interpret file '/content/drive/MyDrive/Colab Notebooks/ground_truth.npy' as a pickle

当设置allow_pickle=False时：

ValueError                                Traceback (most recent call last)
<ipython-input-9-fd12baa39e9b> in <cell line: 1>()
----> 1 a = np.load('/content/drive/MyDrive/Colab Notebooks/ground_truth.npy', encoding='bytes')

/usr/local/lib/python3.10/dist-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding, max_header_size)
    436             # Try a pickle
    437             if not allow_pickle:
--> 438                 raise ValueError("Cannot load file containing pickled data "
    439                                  "when allow_pickle=False")
    440             try:

ValueError: Cannot load the file containing pickled data when allow_pickle=False

所用加载代码 #

import numpy as np
import scipy as sci
from google.colab import drive
drive.mount('/content/drive')

a = np.load('/content/drive/MyDrive/Colab Notebooks/ground_truth.npy', encoding='bytes', allow_pickle=True)

解决方法 #

1. 重新上传文件，排除Drive同步损坏 #

Google Drive同步大文件时可能出现字节损坏，直接上传到Colab会话存储测试：

from google.colab import files
uploaded = files.upload()  # 选择本地的ground_truth.npy

import numpy as np
a = np.load('ground_truth.npy', allow_pickle=True)

如果能正常加载，说明Drive同步有问题，再重新上传该文件到Drive目录。

2. 统一NumPy版本 #

Colab的NumPy版本（1.23.5）和本地（1.24.3）存在差异，升级Colab的NumPy到对应版本：

!pip install --upgrade numpy==1.24.3
import numpy as np
a = np.load('/content/drive/MyDrive/Colab Notebooks/ground_truth.npy', allow_pickle=True, encoding='bytes')

3. 重新生成兼容格式的.npy文件 #

在本地重新保存文件，确保序列化格式兼容旧版本NumPy：

# 本地Jupyter运行
import numpy as np
data = np.load('ground_truth.npy', allow_pickle=True)
np.save('ground_truth_compat.npy', data, allow_pickle=True)

将生成的ground_truth_compat.npy上传到Colab加载。

4. 尝试直接用pickle加载 #

如果文件实际是用pickle而非np.save保存的，直接用pickle读取：

import pickle
with open('/content/drive/MyDrive/Colab Notebooks/ground_truth.npy', 'rb') as f:
    a = pickle.load(f, encoding='bytes')

内容的提问来源于stack exchange，提问作者rtciv