PyTorch中VGG19模型训练Loss无变化问题求助

Hey there, let's figure out why your VGG19 is stuck hovering around that 5.3 loss when training on Tiny ImageNet. I spotted several critical issues in your model architecture and training code that are almost certainly causing this stagnation—let's break them down and fix them step by step:

1. Model Architecture Fixes (Biggest Culprits!) #

- Softmax + CrossEntropyLoss Conflict #

PyTorch's nn.CrossEntropyLoss already combines LogSoftmax and NLLLoss under the hood. Adding a nn.Softmax() at the end of your model means you're passing normalized probabilities into a loss function that expects raw logits. This completely breaks gradient flow, which is why your loss never decreases. Remove the nn.Softmax() layer immediately.

- Wrong Dropout Type & Placement #

nn.Dropout2d() is designed for 2D convolutional feature maps, not flattened fully-connected layer outputs. Use nn.Dropout() instead. Also, placing dropout after Softmax (which we're removing) does nothing useful—move it between fully-connected layers to prevent overfitting.

- Missing ReLU Activations in Classifier #

Your linear layers don't have non-linear ReLU activations between them! Without these, the entire classifier acts like a single linear transformation, severely limiting the model's capacity to learn complex patterns. Add nn.ReLU() after every linear layer except the final one.

Here's the corrected classifier section of your model:

nn.Flatten(),
nn.Linear(25088, 4096),
nn.ReLU(),
nn.Dropout(0.5),  # Use Dropout, not Dropout2d
nn.Linear(4096, 1000),
nn.ReLU(),
nn.Dropout(0.5),
nn.Linear(1000, 200),  # No Softmax here—CrossEntropyLoss handles it

2. Training Code Improvements #

- Stop Repeating model.cuda() #

You're calling model.cuda() every iteration inside the batch loop, which moves the model to GPU repeatedly (unnecessary and error-prone). Move this line once before training starts:

model = model.cuda()

- Ditch Outdated Variable Usage #

In modern PyTorch (v0.4+), Variable is deprecated. Just convert your batches to GPU tensors directly:

X_batch = torch.FloatTensor(X_batch).cuda()
y_batch = torch.LongTensor(y_batch).cuda()

- Lower Your Adam Learning Rate #

Adam works best with small learning rates—0.01 is 10-100x too large. Start with lr=1e-4 (0.0001) and adjust gradually if needed. Your original lr was causing the optimizer to jump around too much, preventing convergence.

- Simplify Loss Logging #

Your current loss calculation averages a huge slice of train_loss, which doesn't reflect recent batch performance well. For clearer visibility, log the current batch loss or a small moving average:

if i % (batch_size*100) == 0:
    print(f"############### iter = {i} / {80000} loss = {loss.item()}")

- Use Implicit Forward Call #

Instead of model.cuda().forward(X_batch), just use model(X_batch)—the forward method is called automatically when you invoke the model on inputs.

3. Additional Checks #

• Input Normalization: VGG models expect images normalized with mean [0.485, 0.456, 0.406] and std [0.229, 0.224, 0.225]. If you're skipping this step, it can slow or block convergence.
• Data Shuffling: Ensure your train_batch_gen shuffles the training data every epoch—fixed data order can cause the model to get stuck on repetitive patterns.
• Tiny ImageNet Size: Tiny ImageNet images are typically 64x64, not 224x224. Resizing is fine, but double-check your preprocessing pipeline isn't distorting data in unexpected ways.

Fixing these issues should get your loss trending downward. Start with the model architecture changes and learning rate adjustment—those are the most impactful fixes.

内容的提问来源于stack exchange，提问作者annaFerdsf