为何PyTorch模型接近损失最小值时波动，Keras模型损失却稳定？
2026-7-4
PyTorch与Keras线性回归模型损失表现差异问题
我在学习PyTorch和Keras基础时，实现了单层线性回归模型，两者损失整体都呈下降趋势，但PyTorch模型在接近损失最小值时会出现明显波动，而Keras模型的损失则保持稳定。训练使用的是经过线性验证的合成数据（IceCreamData.csv）。
Keras实现代码与训练结果
代码
IceCream = pd.read_csv("IceCreamData.csv") x_values = IceCream[["Temperature"]] y_values = IceCream["Revenue"] x_train, x_test, y_train, y_test = train_test_split(x_values, y_values, test_size=0.25) model = tf.keras.Sequential() model.add(tf.keras.layers.Dense(units=1, kernel_initializer=tf.keras.initializers.RandomNormal(stddev=0.01), bias_initializer=tf.keras.initializers.Zeros()) ) model.compile(optimizer=tf.keras.optimizers.Adam(0.01, epsilon=1e-07), loss='mean_squared_error') model.fit(x_train, y_train, epochs=25, batch_size=1)
训练输出
375/375 [==============================] - 0s 617us/step - loss: 261208.2969 Epoch 2/25 375/375 [==============================] - 0s 568us/step - loss: 192060.6094 Epoch 3/25 375/375 [==============================] - 0s 577us/step - loss: 137438.0000 (...) Epoch 20/25 375/375 [==============================] - 0s 536us/step - loss: 667.3316 Epoch 21/25 375/375 [==============================] - 0s 535us/step - loss: 665.7455 Epoch 22/25 375/375 [==============================] - 0s 535us/step - loss: 666.8908 Epoch 23/25 375/375 [==============================] - 0s 577us/step - loss: 665.0857 Epoch 24/25 375/375 [==============================] - 0s 536us/step - loss: 662.0533 Epoch 25/25 375/375 [==============================] - 0s 534us/step - loss: 661.3047
PyTorch实现代码与训练结果
代码
class RegressionDataset(Dataset): def __init__(self, x, y): super().__init__() self.x = torch.from_numpy(x.astype("float32")) self.y = torch.from_numpy(y.astype("float32")) def __len__(self): return len(self.x) def __getitem__(self, index): return self.x[index], self.y[index].unsqueeze(0) class LinearRegressionModel(nn.Module): def __init__(self): super().__init__() self.linear = nn.Linear(1, 1) self.loss_function = nn.MSELoss() self.optimizer_function = torch.optim.Adam(self.parameters(), lr=0.01, eps=1e-07) torch.nn.init.normal_(self.linear.weight, mean=0.0, std=1.0) def forward(self, inputs): return self.linear(inputs) def backward(self, train_loader, epoch, num_epochs): self.train() for x_values, y_values in train_loader: prediction = self.linear(x_values) loss = self.loss_function(prediction, y_values) loss.backward() self.optimizer_function.step() self.optimizer_function.zero_grad() print(f"Epoch [{epoch + 1:03}/{num_epochs:3}] | Train Loss: {loss.item():.4f}") def validate(self, val_loader): self.eval() with torch.no_grad(): for inputs, targets in val_loader: outputs = self.linear(inputs) loss = self.loss_function(outputs, targets) print(f'Validation Loss: {loss.item():.4f}') data = pd.read_csv("./IceCreamData.csv", delimiter=",") x_values = data[["Temperature"]].to_numpy() y_values = data["Revenue"].to_numpy() dataset = RegressionDataset(x_values, y_values) train_dataset, test_dataset = random_split(dataset, lengths=[0.75, 0.25]) train_loader = DataLoader(dataset=train_dataset, batch_size=1, shuffle=True) test_loader = DataLoader(dataset=test_dataset, batch_size=1, shuffle=True) model = LinearRegressionModel() num_epochs = 25 for epoch in range(num_epochs): model.backward(train_loader, epoch, num_epochs) model.validate(test_loader)
训练输出
Epoch [001/ 25] | Train Loss: 248788.2500 Validation Loss: 257732.9062 Epoch [002/ 25] | Train Loss: 96519.7422 Validation Loss: 110466.8281 Epoch [003/ 25] | Train Loss: 76869.0547 Validation Loss: 178772.9375 (...) Epoch [020/ 25] | Train Loss: 679.7694 Validation Loss: 1674.3351 Epoch [021/ 25] | Train Loss: 2065.5454 Validation Loss: 1177.6052 Epoch [022/ 25] | Train Loss: 269.6078 Validation Loss: 595.9854 Epoch [023/ 25] | Train Loss: 115.4116 Validation Loss: 0.1172 Epoch [024/ 25] | Train Loss: 2134.9248 Validation Loss: 9816.9375 Epoch [025/ 25] | Train Loss: 37.1115 Validation Loss: 2869.8569
我已经尝试调整权重初始化（指定标准差）、更换学习率，也对齐了Adam优化器的momentum、betas参数，仅epsilon已调整一致，但问题仍存在。请问为何PyTorch模型接近损失最小值时损失波动，而Keras模型损失稳定？
问题原因分析与解决方法
核心差异点
损失计算与输出的逻辑不同
- Keras的
model.fit输出的是整个epoch所有batch的平均损失，这是对整个训练集的损失统计，所以会呈现平滑下降后稳定的趋势。 - 你的PyTorch代码中，训练阶段最后打印的是最后一个batch的单样本损失值，验证阶段同样只打印最后一个batch的单样本损失。单样本的损失本身就会因为样本差异出现大幅波动，尤其是当模型接近收敛时，不同样本的预测误差差异会被放大，导致输出的数值上下跳变。
- Keras的
权重初始化的差异
你PyTorch代码中权重初始化用的是std=1.0，而Keras中是stddev=0.01，虽然你提到调整了初始化，但看代码里PyTorch的初始化还是std=1.0，这会导致初始权重值过大，训练初期损失波动更剧烈，后续即使收敛也可能因为前期的优化轨迹影响，出现不稳定的情况。
解决步骤
修正PyTorch的损失统计逻辑
在训练和验证阶段，需要累计所有batch的损失，然后计算平均损失再输出：- 训练阶段修改：
def backward(self, train_loader, epoch, num_epochs): self.train() total_loss = 0.0 count = 0 for x_values, y_values in train_loader: prediction = self.linear(x_values) loss = self.loss_function(prediction, y_values) total_loss += loss.item() count += 1 loss.backward() self.optimizer_function.step() self.optimizer_function.zero_grad() avg_loss = total_loss / count print(f"Epoch [{epoch + 1:03}/{num_epochs:3}] | Train Loss: {avg_loss:.4f}") - 验证阶段修改：
def validate(self, val_loader): self.eval() total_loss = 0.0 count = 0 with torch.no_grad(): for inputs, targets in val_loader: outputs = self.linear(inputs) loss = self.loss_function(outputs, targets) total_loss += loss.item() count += 1 avg_loss = total_loss / count print(f'Validation Loss: {avg_loss:.4f}')
- 训练阶段修改：
对齐权重初始化参数
将PyTorch的权重初始化标准差改为和Keras一致的0.01：torch.nn.init.normal_(self.linear.weight, mean=0.0, std=0.01)数据划分的随机性对齐
Keras用train_test_split，PyTorch用random_split，两者的随机种子默认不同，可能导致训练集和测试集分布有差异。可以手动设置相同的随机种子，确保数据划分一致：- Keras侧：
import numpy as np np.random.seed(42) x_train, x_test, y_train, y_test = train_test_split(...) - PyTorch侧：
import torch torch.manual_seed(42) train_dataset, test_dataset = random_split(dataset, lengths=[0.75, 0.25], generator=torch.Generator().manual_seed(42))
- Keras侧：
验证效果
修改后，PyTorch模型输出的训练和验证损失会和Keras一样，是整个epoch的平均损失，波动会大幅降低，接近收敛时会保持稳定。
内容的提问来源于stack exchange，提问作者Tobias Weiland
相关产品推荐
相关产品推荐
