PyTorch回归模型Loss收敛但所有输出值完全相同问题求助

任务与数据集 #

• 首次将神经网络用于回归任务，此前仅使用过CNN和RNN
• 数据集：30000条数据，每条含50个输入特征，需预测14个输出特征，任务形态：输入(30000×50) → 输出(30000×14)

超参数配置 #

input_size = 50
hidden_size = 40
num_epochs = 7
learning_rate =1.00E-03
output_size=15
batch_size=30

异常现象 #

代码运行正常，训练Loss持续收敛，但输出结果不符合预期：所有样本的14维预测特征完全一致，示例如下：

[[  1.3311,   1.0411,   0.9971,  13.6349,  31.4082,  16.5008,   3.2034,
         -26.2985, -26.3108, -22.4322,  24.3007, -26.2376, -26.2337, -26.2369],
        [  1.3311,   1.0411,   0.9971,  13.6349,  31.4082,  16.5008,   3.2034,
         -26.2985, -26.3108, -22.4322,  24.3007, -26.2376, -26.2337, -26.2369],
        [  1.3311,   1.0411,   0.9971,  13.6349,  31.4082,  16.5008,   3.2034,
         -26.2985, -26.3108, -22.4322,  24.3007, -26.2376, -26.2337, -26.2369]]

该异常在训练过程中就已出现，训练时所有样本的输出同步更新，无法理解Loss为何能收敛至较低值。

训练循环代码 #

# 5. Training loop
n_total_steps = len(DS)
n_iterations = -(-n_total_steps // batch_size) # ceiling division
training_loss=[]

loss_fn = nn.MSELoss()

trainloader = torch.utils.data.DataLoader(
                      DS, 
                      batch_size=batch_size, shuffle = True) 
testloader = torch.utils.data.DataLoader(
                      TS,
                      batch_size=batch_size)
  
for epoch in range(num_epochs):
    print('\n')

    for i, (data, target) in enumerate(trainloader): 
        data, target = data.to(device), target.to(device)
        outputs = model(data)
        loss = torch.sqrt(loss_fn(outputs, target))
        training_loss.append(loss.item())

        # 5.5 Backward pass
        opt.zero_grad() # 5.6 Empty the values in the gradient attribute, or model.zero_grad()
        loss.backward() # 5.7 Backprop
        opt.step() # 5.8 Update params

        # 5.9 Print loss
        if (i+1) % 100 == 0:
            print(f'Epoch {epoch+1}/{num_epochs}, Iteration {i+1}/{n_iterations}, Loss={loss.item():.4f} ')

训练过程输出示例 #

Epoch 1/7, Iteration 100/1321, Loss=1.5157 
tensor([[  1.4186,   1.1157,   1.0471,  13.5818,  31.3844,  16.5334,   3.1015,
         -26.3141, -26.2974, -22.4117,  24.3678, -26.2477, -26.2577, -26.2387],
        [  1.4186,   1.1157,   1.0471,  13.5818,  31.3844,  16.5334,   3.1015,
         -26.3141, -26.2974, -22.4117,  24.3678, -26.2477, -26.2577, -26.2387],
        [  1.4186,   1.1157,   1.0471,  13.5818,  31.3844,  16.5334,   3.1015,
         -26.3141, -26.2974, -22.4117,  24.3678, -26.2477, -26.2577, -26.2387],
        ...
Epoch 1/7, Iteration 300/1321, Loss=0.9697 
tensor([[  1.3142,   1.0427,   0.9661,  13.6267,  31.2973,  16.5265,   3.1028,
         -26.2207, -26.2468, -22.3516,  24.4410, -26.1698, -26.1708, -26.1715],
        [  1.3142,   1.0427,   0.9661,  13.6267,  31.2973,  16.5265,   3.1028,
         -26.2207, -26.2468, -22.3516,  24.4410, -26.1698, -26.1708, -26.1715],
        ...

模型代码 #

class MyDataset(Dataset) :

    def __init__(self, file_name) :
        train_df = pd.read_csv(file_name)
        x = train_df.filter(regex='X') # Input : X Featrue
        y = train_df.filter(regex='Y') # Output : Y Feature
        self.train_x = torch.tensor(x.values,dtype=torch.float32)
        self.train_y = torch.tensor(y.values,dtype=torch.float32)

    def __len__(self) :
        return len(self.train_y)

    def __getitem__ (self,idx) :
        return self.train_x[idx],self.train_y[idx]
  
class LGNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.layer1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.Tanh()
        self.layer2 = nn.Linear(hidden_size, hidden_size)
        self.layer3 = nn.Linear(hidden_size, hidden_size)
        self.layer4 = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        out = self.layer1(x)
        out = self.relu(out)
        out = self.layer2(out)
        out = self.relu(out)
        out = self.layer3(out)
        out = self.relu(out)
        out = self.layer4(out)
        return out

# 4.1 Create NN model instance
model = LGNN(input_size, hidden_size, output_size).to(device) #to(device)는 GPU
model.apply(reset_weights)
# 4.2 Loss and Optimiser
opt = optim.Adam(model.parameters(), lr=learning_rate)
loss_fn = nn.MSELoss()

排查情况 #

已进行k折验证，未发现过拟合问题。

原因分析与解决方法 #

可能原因 #

1. 输出维度不匹配：超参数中output_size=15，但实际需要预测14个输出特征，维度不匹配干扰模型学习逻辑。
2. 激活函数导致梯度消失：连续3层使用Tanh激活函数，易引发梯度消失，使得模型参数更新停滞，最终所有样本输出收敛到全局均值。
3. 数据未标准化：输入/目标特征尺度差异过大，模型倾向于学习全局均值而非样本间的差异化特征。
4. 权重初始化不当：reset_weights函数可能存在初始化错误，导致神经元输出一致，后续层无法学习样本差异。

解决方法 #

1. 修正输出维度：将output_size改为14，与实际预测特征数量匹配。
2. 调整激活函数与网络深度：替换Tanh为ReLU/LeakyReLU，减少梯度消失概率；可适当减少隐藏层数量，简化网络结构。
3. 标准化数据：对输入特征X和目标特征Y做Z-score标准化，消除尺度差异，让模型聚焦于样本间的差异学习。
4. 优化权重初始化：确保reset_weights使用合理的初始化策略，例如：

   def reset_weights(m):
       if isinstance(m, nn.Linear):
           nn.init.xavier_uniform_(m.weight)
           nn.init.zeros_(m.bias)
   

5. 调整训练参数：尝试将学习率调整为5e-4或2e-3，同时增加epoch数，给模型足够的学习时间。

内容的提问来源于stack exchange，提问作者Jinju Kim