为何Gensim Word2Vec模型训练时每轮损失持续上升？

问题描述 #

使用Gensim的Word2Vec在Twitter数据上训练词向量模型，发现模型损失在每个epoch中持续上升，第一轮损失最低。

相关代码 #

loss_list = []
class callback(CallbackAny2Vec):
     
    def __init__(self):
        self.epoch = 0
          
    def on_epoch_end(self, model):
        loss = model.get_latest_training_loss()
        loss_list.append(loss)
        print('Loss after epoch {}: {}'.format(self.epoch, loss))
        self.epoch = self.epoch + 1

model = Word2Vec(df['tweet_text'], vector_size=300, window=10, epochs=30, hs=0, negative = 1, compute_loss=True, callbacks=[callback()])
embedding_size = model.wv.vectors.shape[1]
print("embedding size--->", embedding_size)
vocab = model.wv.index_to_key
print("minimum loss {} at epoch {}".format(min(loss_list), loss_list.index(min(loss_list))))

输出结果 #

Loss after epoch 0: 527066.375
Loss after epoch 1: 1038087.0625
Loss after epoch 2: 1510719.75
Loss after epoch 3: 1936163.875
Loss after epoch 4: 2364015.5
Loss after epoch 5: 2779299.75
Loss after epoch 6: 3183956.25
Loss after epoch 7: 3570054.5
Loss after epoch 8: 3966524.75
Loss after epoch 9: 4335994.5
Loss after epoch 10: 4706316.0
Loss after epoch 11: 5046213.0
Loss after epoch 12: 5410604.5
Loss after epoch 13: 5754962.0
Loss after epoch 14: 6080469.0
Loss after epoch 15: 6428622.5
Loss after epoch 16: 6771707.0
Loss after epoch 17: 7105302.0
Loss after epoch 18: 7400089.0
Loss after epoch 19: 7732032.0
Loss after epoch 20: 8059942.5
Loss after epoch 21: 8408386.0
Loss after epoch 22: 8685176.0
Loss after epoch 23: 8959723.0
Loss after epoch 24: 9242788.0
Loss after epoch 25: 9506676.0
Loss after epoch 26: 9752588.0
Loss after epoch 27: 10013168.0
Loss after epoch 28: 10288152.0
Loss after epoch 29: 10550915.0
embedding size---> 300
minimum loss 527066.375 at epoch 0

原因分析 #

1. 累计损失而非单轮损失
   Gensim的get_latest_training_loss()返回的是从训练开始到当前epoch的累计总损失，不是单轮epoch的损失。所以你看到的数值持续上升是正常的累计结果，并非每轮损失真的在增加。要查看单轮损失，需要用当前epoch的累计值减去上一轮的累计值，比如epoch 1的单轮损失为1038087.0625 - 527066.375，这样才能判断每轮损失的变化趋势。

2. 负采样参数设置不合理
   你设置了negative=1，负采样的样本数量过少。Word2Vec负采样通常建议设置为5-20，过小的负采样数量会导致训练信号不足，模型学习不稳定。

3. 输入数据格式可能有误
   Word2Vec要求输入是分词后的单词列表（每个样本是一个由单词组成的列表），如果df['tweet_text']直接传入的是原始字符串，Gensim会将每个字符当作一个独立的词，这会导致词汇量异常庞大，训练逻辑完全偏离预期，也可能引发损失异常。需要确认输入数据是否已经完成正确的分词处理。

4. 训练轮次可能过多
   设置了epochs=30，对于Word2Vec来说这个轮次可能偏多，尤其是当数据集规模不大时，容易出现过拟合。不过这不是当前损失持续上升的直接原因，但会影响最终模型效果。

内容的提问来源于stack exchange，提问作者Debbie