PyTorch LSTM回归模型任意输入均输出相同值的原因及解决办法

数据集说明 #

左侧为输入，右侧为输出。输入经分词后转换为索引列表，例如分子输入CC1(C)Oc2ccc(cc2C@HN3CCCC3=O)C#N会被转换为如下序列：

[28, 28, 53, 69, 28, 70, 40, 2, 54, 2, 2, 2, 69, 2, 2, 54, 67, 28, 73, 33, 68, 69, 67, 28, 73, 73, 33, 68, 53, 40, 70, 39, 55, 28, 28, 28, 28, 55, 62, 40, 70, 28, 63, 39, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

使用如下字符列表作为字符串到索引的映射：

cs = ['a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z',    'A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z','0','1','2','3','4','5','6','7','8','9','=','#',':','+','-','[',']','(',')','/','\'', '@','.','%']

输入字符串的每个字符对应一个索引，若字符串长度小于所有输入的最大长度100，则补0（如上例所示）。

模型代码 #

import torch
from torch import nn
import torch.nn.functional as F

class LSTM_regr(torch.nn.Module) :
    def __init__(self, vocab_size, embedding_dim, hidden_dim) :
        super().__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True)
        self.linear = nn.Linear(hidden_dim, 1)
        self.dropout = nn.Dropout(0.2)  
        
    def forward(self, x, l):
        x = self.embeddings(x)
        x = self.dropout(x)
        lstm_out, (ht, ct) = self.lstm(x)
        return self.linear(ht[-1])

vocab_size = 76
model =  LSTM_regr(vocab_size, 20, 256)

问题描述 #

训练完成后，输入任意测试样本，模型都返回相同输出（例如3.3318）。

训练循环代码 #

def train_model_regr(model, epochs=10, lr=0.001):
    parameters = filter(lambda p: p.requires_grad, model.parameters())
    optimizer = torch.optim.Adam(parameters, lr=lr)
    for i in range(epochs):
        model.train()
        sum_loss = 0.0
        total = 0
        for x, y, l in train_dl:
            x = x.long()
            y = y.float()
            y_pred = model(x, l)
            optimizer.zero_grad()
            loss = F.mse_loss(y_pred, y.unsqueeze(-1))
            loss.backward()
            optimizer.step()
            sum_loss += loss.item()*y.shape[0]
            total += y.shape[0]

解决方案 #

将学习率从0.01降低到0.0005，同时将batch size从100调整为10后，模型运行正常。
问题原因：之前batch size过大，模型为了最小化损失，直接学习输出标签的均值，因此出现所有输入返回相同值的情况。

内容的提问来源于stack exchange，提问作者mac179