将类移入模块后,如何正确从joblib加载已保存的类实例?
问题描述
原本在diseaseTree.py文件中定义了DiseaseTree类,并用joblib保存了该类的一个实例。现在重构为模块化结构,目录如下:
/ /models __init__.py models.py diseaseTree.py /plots __init__.py plots.py __init__.py main.py
models/__init__.py内容:
from .models import DatabaseConnector from .diseaseTree import DiseaseTree
在plots/plots.py中执行导入与加载:
from models import DatabaseConnector from models import DiseaseTree import joblib with open("/data/projects/classifiers/data/diseaseTree.joblib", "rb") as file: mainTree = joblib.load(file)
导入操作正常,但加载joblib文件时出现错误:
ModuleNotFoundError: No module named 'diseaseTree'
问题根源是当初保存实例时,pickle记录的类模块路径为diseaseTree,但重构后类的实际路径变为models.diseaseTree。目前可通过临时添加sys.modules绕过,但该方式不优雅,违背模块化初衷,需更合理的解决方法。
优雅解决方案
方法1:重新保存实例(推荐)
这是最直接且无后遗症的方案,通过更新实例的类路径记录彻底解决问题:
import joblib import sys import types from models import DiseaseTree # 临时加载旧实例 temp_module = types.ModuleType('diseaseTree') temp_module.DiseaseTree = DiseaseTree sys.modules['diseaseTree'] = temp_module with open("/data/projects/classifiers/data/diseaseTree.joblib", "rb") as f: old_tree = joblib.load(f) # 清理临时模块 del sys.modules['diseaseTree'] # 用当前类路径重新保存实例 with open("/data/projects/classifiers/data/diseaseTree_updated.joblib", "wb") as f: joblib.dump(old_tree, f)
之后直接加载新文件即可,无需额外操作:
import joblib with open("/data/projects/classifiers/data/diseaseTree_updated.joblib", "rb") as f: mainTree = joblib.load(f)
方法2:自定义Unpickler映射类路径
如果无法修改旧的joblib文件,可通过自定义Unpickler在加载时修正类的模块路径:
import joblib import pickle from models import DiseaseTree class CustomUnpickler(pickle.Unpickler): def find_class(self, module, name): # 将旧模块名映射到新路径 if module == 'diseaseTree' and name == 'DiseaseTree': return DiseaseTree return super().find_class(module, name) # 使用自定义Unpickler加载实例 with open("/data/projects/classifiers/data/diseaseTree.joblib", "rb") as f: mainTree = CustomUnpickler(f).load()
该方法无需修改原文件,也不需要临时模块,直接在加载过程中完成路径映射。
方法3:在模块中添加兼容别名
通过在models模块中注册旧模块名的别名,让pickle能自动找到对应类:
在models/__init__.py中添加以下代码:
import sys from .diseaseTree import DiseaseTree # 将models.diseaseTree注册为全局模块中的'diseaseTree' sys.modules['diseaseTree'] = sys.modules['models.diseaseTree']
这样pickle在寻找diseaseTree.DiseaseTree时,会自动指向models.diseaseTree.DiseaseTree,无需临时模块操作。不过该方式会在全局模块空间留下别名,需根据模块化设计需求权衡使用。
内容的提问来源于stack exchange,提问作者Ben
相关产品推荐
相关产品推荐

