保存含Keras模型的变量列表遇h5py报错,需实现压缩保存
解决Keras模型列表压缩保存的问题
问题原因
直接用pickle+gzip保存Keras模型对象时,pickle序列化模型的过程中会触发Keras底层的模型保存逻辑,该逻辑依赖h5py。即便你已安装h5py,也可能因为Python环境不匹配(比如h5py安装在其他虚拟环境)导致报错。此外,直接pickle模型对象本身也不是Keras推荐的保存方式。
可行解决方案
方案1:单独保存模型为HDF5后打包压缩
先将每个模型保存为标准HDF5文件,再把所有HDF5文件打包压缩,这种方式兼容性最好:
保存代码
import gzip import os from keras.models import load_model # 临时保存每个模型 temp_model_paths = [] for idx, model in enumerate(list_of_vars): temp_path = f"temp_model_{idx}.h5" model.save(temp_path) temp_model_paths.append(temp_path) # 将所有临时模型文件写入压缩包 with gzip.open("compressed_models.gz", "wb") as zip_handle: for path in temp_model_paths: with open(path, "rb") as model_handle: zip_handle.write(model_handle.read()) # 清理临时文件 for path in temp_model_paths: os.remove(path)
加载代码
import gzip import os from keras.models import load_model # 从压缩包提取临时模型文件 temp_model_paths = [] with gzip.open("compressed_models.gz", "rb") as zip_handle: for idx in range(len(list_of_vars)): temp_path = f"temp_model_{idx}.h5" temp_model_paths.append(temp_path) with open(temp_path, "wb") as model_handle: model_handle.write(zip_handle.read()) # 加载模型并清理临时文件 loaded_models = [] for path in temp_model_paths: model = load_model(path) loaded_models.append(model) os.remove(path)
方案2:分离模型结构与权重后序列化保存
只保存模型的结构(JSON格式)和权重数组,再用pickle+gzip压缩,避免直接序列化模型对象:
保存代码
import gzip import pickle from keras.models import model_from_json # 提取每个模型的结构和权重 model_info_list = [] for model in list_of_vars: model_json = model.to_json() model_weights = model.get_weights() model_info_list.append((model_json, model_weights)) # 压缩保存结构与权重数据 with gzip.open("models_struct_weights.pkl.gz", "wb") as handle: pickle.dump(model_info_list, handle)
加载代码
import gzip import pickle from keras.models import model_from_json # 加载并解压数据 with gzip.open("models_struct_weights.pkl.gz", "rb") as handle: model_info_list = pickle.load(handle) # 重建模型 loaded_models = [] for json_str, weights in model_info_list: model = model_from_json(json_str) model.set_weights(weights) loaded_models.append(model)
方案3:修复h5py环境问题
如果坚持要用原方法,先确认当前Python环境是否真的安装了h5py:
- 运行以下代码检查:
import h5py print(h5py.__version__)
- 如果报错,在当前环境重新安装h5py:
# pip环境 pip install h5py # conda环境 conda install h5py
内容的提问来源于stack exchange,提问作者Morteza
相关产品推荐
相关产品推荐

