Google Colab加载NLP项目pickle文件遇int64 dtype解析错误

问题背景 #

运行adashofdata的NLP项目时，在Google Colab环境执行4-Topic-Modeling.ipynb代码，加载dtm_stop.pkl文件时触发TypeError: Cannot interpret 'dtype('int64')' as a data type错误，已尝试pickle、cloudpickle、joblib加载均无效，需要可行的解决或格式转换方案。

报错代码 #

import pickle
import pandas as pd
import numpy as np
from google.colab import drive
import joblib

# 挂载Google Drive
drive.mount('/content/drive')

# 文件路径
file_path = '/content/drive/MyDrive/Colab Notebooks/pynb/pickle/'

# 尝试加载文件
try:
    with open(file_path + 'dtm_stop.pkl', 'rb') as f:
        data = joblib.load(file_path + 'dtm_stop.pkl')  # 使用joblib加载

    # 检查加载数据的类型
    print("Data type:", type(data))

    # 转换数据类型处理错误值
    if isinstance(data, np.ndarray):
        print("Data loaded as numpy ndarray")
        data = data.astype(np.float64, errors='ignore')
    elif isinstance(data, pd.DataFrame):
        print("Data loaded as DataFrame")
        data = data.apply(pd.to_numeric, errors='coerce')
    else:
        print("Data is not a DataFrame or numpy array, data type:", type(data))

except Exception as e:
    print(f"Error loading with pickle: {e}")
    raise e

# 输出加载的数据
print(data)

错误信息 #

Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
Error loading with pickle: Cannot interpret 'dtype('int64')' as a data type
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-22-285aaeee691b> in <cell line: 0>()
     30 except Exception as e:
     31     print(f"Error loading with pickle: {e}")  # Print the error message in more detail
---> 32     raise e
     33 
     34 # Let's see the loaded data

5 frames
/usr/local/lib/python3.11/dist-packages/numpy/_core/numeric.py in _frombuffer(buf, dtype, shape, order)

TypeError: Cannot interpret 'dtype('int64')' as a data type

解决方法 #

1. 匹配原项目依赖版本 #

该错误大概率是Colab默认的numpy/scikit-learn版本与原项目保存dtm_stop.pkl时的版本不兼容导致。查看原项目的requirements.txt或环境配置，在Colab中安装对应版本的依赖：

# 示例：替换为原项目实际使用的版本号
!pip install numpy==1.21.6 pandas==1.3.5 scikit-learn==1.0.2

安装完成后重启Colab内核，再尝试加载文件。

2. 调整加载参数 #

尝试使用pickle低协议加载，或指定编码格式：

import pickle

with open(file_path + 'dtm_stop.pkl', 'rb') as f:
    # 使用协议4加载（适配旧版本序列化的文件）
    data = pickle.load(f, protocol=4, encoding='latin1')

或使用cloudpickle加载：

import cloudpickle

with open(file_path + 'dtm_stop.pkl', 'rb') as f:
    data = cloudpickle.load(f)

3. 转换文件格式（替代方案） #

如果无法通过版本或加载参数解决，可在本地能正常加载该pkl文件的环境中，将其转换为兼容性更好的格式（如Parquet、NPZ），再上传到Colab使用：

本地转换代码：

import pickle
import pandas as pd
import numpy as np

# 本地加载pkl文件
with open('dtm_stop.pkl', 'rb') as f:
    dtm_data = pickle.load(f)

# 根据数据类型选择转换格式
if isinstance(dtm_data, pd.DataFrame):
    # 转Parquet格式（高效且保留类型信息）
    dtm_data.to_parquet('dtm_stop.parquet')
elif isinstance(dtm_data, np.ndarray):
    # 转NPZ格式（numpy专用格式）
    np.savez('dtm_stop.npz', data=dtm_data)

Colab加载转换后的文件：

# 加载Parquet文件
import pandas as pd
dtm_data = pd.read_parquet('/content/drive/MyDrive/Colab Notebooks/pynb/pickle/dtm_stop.parquet')

# 或加载NPZ文件
import numpy as np
loaded_data = np.load('/content/drive/MyDrive/Colab Notebooks/pynb/pickle/dtm_stop.npz')
dtm_data = loaded_data['data']

4. 重新生成pkl文件 #

直接在Colab中从头运行项目的前置notebook（如1-Data-Cleaning.ipynb到3-Text-Representation.ipynb），重新生成dtm_stop.pkl文件，完全规避版本差异带来的兼容性问题。

内容的提问来源于stack exchange，提问作者mymiracl