PyTorch训练Fashion-MNIST时损失上升问题求助

Hey there! Let's break down why your training loss is climbing instead of falling—especially since more complex networks seem to make this issue worse. Based on typical PyTorch pitfalls with this dataset, here are key areas to check in your main.py code:

• Overly high learning rate
  This is the most common culprit. If your learning rate is too large, model parameters update in huge jumps, causing the optimizer to overshoot the loss function's minimum and bounce further away. Complex networks have more parameters, so this instability gets amplified. Try scaling down your learning rate drastically—start with 0.001 instead of 0.1, for example, and adjust from there.

• Missing data normalization
  Fashion-MNIST pixel values range from 0 to 255. Feeding raw, unnormalized data into your network can lead to unstable weight updates, as large input values cause extreme gradients. Add a normalization step to your data transforms, like:

  transforms.Normalize((0.2860,), (0.3530,))  # Using Fashion-MNIST's actual mean/std
  

  This scales pixels to a range that’s easier for the model to learn from.

• Poor weight initialization
  Complex networks are extra sensitive to bad initial weights. If weights start too large, activation functions (like Sigmoid or ReLU) can hit saturation points, leading to vanishing or exploding gradients. Try using targeted initialization methods:

  # For a linear layer, use Xavier initialization
  torch.nn.init.xavier_uniform_(your_linear_layer.weight)
  

  PyTorch’s default initialization might not cut it for deeper networks.

• Mismatched loss function/optimizer setup
  Double-check your loss function: Fashion-MNIST is a 10-class classification task, so you should use CrossEntropyLoss (note: this loss includes a Softmax layer, so don’t add a Softmax to your network’s final layer). For optimizers, adding momentum to SGD can stabilize training:

  optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
  

• Training loop bugs
  Quick sanity checks here:

  • Are you calling optimizer.zero_grad() at the start of each batch? If not, gradients accumulate across batches and cause broken updates.
  • Did you set your model to training mode with model.train() before starting training?
  • Are you passing inputs and labels to the loss function in the correct order?

Start with the simplest fixes first (normalization + lower learning rate) and see if your loss starts decreasing. If not, dig into the weight initialization and training loop details next.

内容的提问来源于stack exchange，提问作者Seb