Absolutely, you can dynamically tweak your optimizer's learning rate at any point during training without relying on PyTorch's built-in LR schedulers. Here's a straightforward solution, using your SGD example as a reference:

Key Background: Optimizer param_groups #

Every PyTorch optimizer stores its configuration in the param_groups attribute—a list of dictionaries. Each dictionary contains settings (like lr, weight_decay, etc.) for a subset of your model's parameters. Even if you didn't explicitly split parameters into groups, there will always be at least one group in this list.

Step-by-Step Implementation #

Let's start with your example initialization:

import torch
import torch.nn as nn
from torch.optim import SGD

# Sample model and optimizer
model = nn.Linear(10, 2)
optim = SGD(model.parameters(), lr=0.01)

1. Update Learning Rate for All Parameter Groups #

If you want to apply the same new LR to all model parameters, loop through each group in param_groups and set the lr key to your desired value:

# Set the new learning rate
new_lr = 0.001
for param_group in optim.param_groups:
    param_group['lr'] = new_lr

2. Verify the Update #

To confirm the change took effect, print the LR value from the first parameter group (since you only have one here):

print(optim.param_groups[0]['lr'])  # Output: 0.001

3. Target Specific Parameter Groups (Advanced) #

If you’ve split your model into multiple parameter groups (e.g., different LRs for the backbone and classification head), you can update just one group by its index. For example:

# Initialize optimizer with two parameter groups
optim = SGD([
    {'params': model.backbone.parameters(), 'lr': 0.01},
    {'params': model.head.parameters(), 'lr': 0.001}
])

# Only update the backbone's learning rate
optim.param_groups[0]['lr'] = 0.0005

Important Notes #

• This approach works for all PyTorch optimizers (Adam, RMSprop, Adagrad, etc.)—they all use the param_groups structure.
• You can adjust the LR at any point in your training loop: after a fixed number of epochs, based on validation performance metrics, or manually if you notice the current LR is too high/low (like in your example).
• Unlike pre-defined schedulers, this method gives you full, on-demand control to set the LR to any value you want, no prior planning required.

内容的提问来源于stack exchange，提问作者patapouf_ai