使用Gensim训练Word2Vec模型时损失先升后降的原因探究

这是此前相关问题的后续。当前使用Gensim训练Word2Vec模型，参数设置为hs=1、sg=0、negative=0，修改代码后训练时间缩短，但损失值出现先上升后下降的异常趋势，需明确背后原因。

训练代码 #

from gensim.models.keyedvectors import KeyedVectors
from gensim.models import word2vec
import logging

logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', 
level=logging.INFO)
sentences = word2vec.Text8Corpus("text8")   # 加载语料库


from gensim.models.callbacks import CallbackAny2Vec
loss_list = []
    
class Callback(CallbackAny2Vec):
    def __init__(self):
        self.epoch = 0

    def on_epoch_end(self, model):
        loss = model.get_latest_training_loss()
        loss_list.append(loss)
        print('Loss after epoch {}:{}'.format(self.epoch, loss))
        model.running_training_loss = 0.0
        self.epoch = self.epoch + 1


from gensim.models import KeyedVectors,word2vec,Word2Vec
import time

start_time = time.time()

model = word2vec.Word2Vec(sentences, hs=1, sg=0, negative=0, compute_loss=True, epochs=30, callbacks=[Callback()])

end_time = time.time()

print('Running time: %s seconds' % (end_time - start_time))

训练输出详情 #

Loss after epoch 0:39370848.0
Loss after epoch 1:43579636.0
Loss after epoch 2:45213772.0
Loss after epoch 3:46132356.0
Loss after epoch 4:46788412.0
Loss after epoch 5:47218508.0
Loss after epoch 6:47553520.0
Loss after epoch 7:47793332.0
Loss after epoch 8:47995616.0
Loss after epoch 9:48134664.0
Loss after epoch 10:48224960.0
Loss after epoch 11:48326640.0
Loss after epoch 12:48371072.0
Loss after epoch 13:48405980.0
Loss after epoch 14:48437804.0
Loss after epoch 15:48417612.0
Loss after epoch 16:48415112.0
Loss after epoch 17:48396260.0
Loss after epoch 18:48349064.0
Loss after epoch 19:48301088.0
Loss after epoch 20:48247328.0
Loss after epoch 21:48167340.0
Loss after epoch 22:48053500.0
Loss after epoch 23:47937300.0
Loss after epoch 24:47810964.0
Loss after epoch 25:47669088.0
Loss after epoch 26:47500524.0
Loss after epoch 27:47300488.0
Loss after epoch 28:47044920.0
Loss after epoch 29:46747080.0

Running time: 259.9046218395233 seconds

原因分析 #

1. 损失计算的累积逻辑偏差 #

回调函数中每个epoch结束后手动重置model.running_training_loss为0.0，但Gensim的get_latest_training_loss()返回的是上次重置以来的累积损失。旧版本Gensim中，分层softmax（hs=1）的损失累积逻辑存在初始化偏差：

• 第一个epoch的损失是真实训练损失；
• 后续epoch开始时，参数已更新，但损失累加器的重置时机与参数更新不同步，导致前几个epoch的损失被错误累加，出现数值上升的假象。

2. 分层softmax的训练特性 #

采用sg=0（CBOW模式）+hs=1（分层softmax）时，模型目标是预测中心词，通过哈夫曼树计算损失。初始阶段参数随机初始化，对语料拟合能力弱，哈夫曼树路径计算产生更多错误预测，导致损失暂时上升；随着训练推进，参数优化，预测准确率提升，损失逐步下降。

3. 学习率的影响 #

Gensim默认学习率从0.025线性衰减到0.0001。初始阶段学习率高，参数更新幅度大，可能引发损失短期波动上升；随着学习率降低，参数更新趋于稳定，损失开始下降。

验证与解决建议 #

• 升级Gensim版本：新版本修复了旧版中损失计算的bug；
• 调整损失记录方式：不要手动重置running_training_loss，改为记录当前总损失与上一epoch总损失的差值，得到单epoch真实损失；
• 聚焦模型实际效果：损失趋势仅作参考，最终通过词向量语义相似度、类比任务等验证模型质量。

内容的提问来源于stack exchange，提问作者Evan Parker