加载预训练Word2Vec模型遇UnpicklingError: could not find MARK问题求助

问题概述 #

加载预训练Word2Vec模型时触发UnpicklingError: could not find MARK，此前模型可正常运行。无法重新保存模型，已尝试卸载并重装gensim==3.4.0，问题仍未解决。

加载代码 #

base_model = Word2Vec.load("sample.model", mmap='r')

完整错误栈 #

AttributeError                            Traceback (most recent call last)
File ~/Documents/Docs/lib/python3.9/site-packages/gensim/models/word2vec.py:975, in Word2Vec.load(cls, *args, **kwargs)
    974 try:
--> 975     return super(Word2Vec, cls).load(*args, **kwargs)
    976 except AttributeError:

File ~/Documents/Docs//lib/python3.9/site-packages/gensim/models/base_any2vec.py:629, in BaseWordEmbeddingsModel.load(cls, *args, **kwargs)
    627 @classmethod
    628 def load(cls, *args, **kwargs):
--> 629     model = super(BaseWordEmbeddingsModel, cls).load(*args, **kwargs)
    630     if model.negative and hasattr(model.wv, 'index2word'):

File ~/Documents/Docs/lib/python3.9/site-packages/gensim/models/base_any2vec.py:278, in BaseAny2VecModel.load(cls, fname_or_handle, **kwargs)
    276 @classmethod
    277 def load(cls, fname_or_handle, **kwargs):
--> 278     return super(BaseAny2VecModel, cls).load(fname_or_handle, **kwargs)

File ~/Documents/Docs/lib/python3.9/site-packages/gensim/utils.py:426, in SaveLoad.load(cls, fname, mmap)
    425 obj = unpickle(fname)
--> 426 obj._load_specials(fname, mmap, compress, subname)
    427 logger.info("loaded %s", fname)

File ~/Documents/Docs/lib/python3.9/site-packages/gensim/utils.py:491, in SaveLoad._load_specials(self, fname, mmap, compress, subname)
    490 logger.info("setting ignored attribute %s to None", attrib)
--> 491 setattr(self, attrib, None)

File ~/Documents/Docs/lib/python3.9/site-packages/gensim/utils.py:1398, in deprecated.<locals>.decorator.<locals>.new_func1(*args, **kwargs)
   1393 warnings.warn(
   1394     fmt.format(name=func.__name__, reason=reason),
   1395     category=DeprecationWarning,
   1396     stacklevel=2
   1397 )
--> 1398 return func(*args, **kwargs)

File ~/Documents/Docs/lib/python3.9/site-packages/gensim/models/base_any2vec.py:450, in BaseWordEmbeddingsModel.cum_table(self, value)
    447 @cum_table.setter
    448 @deprecated("Attribute will be removed in 4.0.0, use self.vocabulary.cum_table instead")
    449 def cum_table(self, value):
--> 450     self.vocabulary.cum_table = value

AttributeError: 'Word2Vec' object has no attribute 'vocabulary'

During handling of the above exception, another exception occurred:

UnpicklingError                           Traceback (most recent call last)
Input In [19], in <cell line: 6>()
      3 old_messages = old_messages[['trigramed_tokenized', 'dup_id']].drop_duplicates()
      5 distance = pd.read_csv('distances.csv')
----> 6 base_model = Word2Vec.load("sample.model", mmap='r')
      8 #checking if messages are string
      9 if type(new_message_feats.trigramed_tokenized[0]) == str:

File ~/Documents/Docs/lib/python3.9/site-packages/gensim/models/word2vec.py:979, in Word2Vec.load(cls, *args, **kwargs)
    977 logger.info('Model saved using code from earlier Gensim Version. Re-loading old model in a compatible way.')
    978 from gensim.models.deprecated.word2vec import load_old_word2vec
--> 979 return load_old_word2vec(*args, **kwargs)

File ~/Documents/Docs/lib/python3.9/site-packages/gensim/models/deprecated/word2vec.py:153, in load_old_word2vec(*args, **kwargs)
    152 def load_old_word2vec(*args, **kwargs):
--> 153     old_model = Word2Vec.load(*args, **kwargs)
    154     params = {
    155         'size': old_model.vector_size,
    156         'alpha': old_model.alpha,
   (...)
    173         'compute_loss': old_model.__dict__.get('compute_loss', None)
    174     }
    175     new_model = NewWord2Vec(**params)

File ~/Documents/Docs/lib/python3.9/site-packages/gensim/models/deprecated/word2vec.py:1616, in Word2Vec.load(cls, *args, **kwargs)
   1614 @classmethod
   1615 def load(cls, *args, **kwargs):
--> 1616     model = super(Word2Vec, cls).load(*args, **kwargs)
   1617     # update older models
   1618     if hasattr(model, 'table'):

File ~/Documents/Docs/lib/python3.9/site-packages/gensim/models/deprecated/old_saveload.py:87, in SaveLoad.load(cls, fname, mmap)
     83 logger.info("loading %s object from %s", cls.__name__, fname)
     85 compress, subname = SaveLoad._adapt_by_suffix(fname)
--> 87 obj = unpickle(fname)
     88 obj._load_specials(fname, mmap, compress, subname)
     89 logger.info("loaded %s", fname)

File ~/Documents/Docs/lib/python3.9/site-packages/gensim/models/deprecated/old_saveload.py:380, in unpickle(fname)
    377 file_bytes = file_bytes.replace(
    378     b'gensim.models.wrappers.fasttext', b'gensim.models.deprecated.fasttext_wrapper')
    379 if sys.version_info > (3, 0):
--> 380     return _pickle.loads(file_bytes, encoding='latin1')
    381 else:
    382     return _pickle.loads(file_bytes)

UnpicklingError: could not find MARK

解决办法 #

1. 校验模型文件完整性 #

UnpicklingError多由模型文件损坏引发，先做以下检查：

• 确认sample.model及其关联文件（如.syn1neg.npy、.wv.vectors.npy等）均存在且未被修改
• 用哈希校验工具（如md5sum）对比文件原始哈希值，验证文件未损坏

2. 匹配原始保存模型的Gensim版本 #

若安装3.4.0仍报错，可能原始模型由更早版本保存，尝试降级到指定版本：

pip uninstall gensim -y
pip install gensim==2.3.0

安装完成后重新尝试加载模型。

3. 绕过Gensim加载逻辑，直接读取词向量 #

若仅需使用词向量，可直接读取向量文件重建可用对象：

import numpy as np
from gensim.models.keyedvectors import KeyedVectors

# 读取词汇表文件
with open("sample.model.vocab", "r", encoding="utf-8") as f:
    vocab = [line.strip().split()[0] for line in f.readlines()]

# 读取向量文件
vectors = np.load("sample.model.wv.vectors.npy")

# 构建KeyedVectors对象
kv = KeyedVectors(vector_size=vectors.shape[1])
kv.add_vectors(vocab, vectors)

# 后续用kv["your_word"]获取对应向量

4. 禁用mmap加载模式 #

尝试移除mmap='r'参数，改用常规加载方式：

base_model = Word2Vec.load("sample.model")

内容的提问来源于stack exchange，提问作者mjoy