Nvidia DIGITS学习率歧义：UI基础学习率与Caffe Prototxt层lr_pm差异及优先级

Great question—this is a super common point of confusion when mixing DIGITS' user-friendly interface with Caffe's underlying configuration. Let me break down the differences and how they work together clearly:

1. NVIDIA DIGITS' Base Learning Rate #

• This is a global baseline value you set directly in the DIGITS training setup UI. Think of it as the "starting point" for all learning rate calculations across your entire network.
• Critically, this value overrides any base_lr you might have hardcoded in your Caffe solver.prototxt file. If you set it in DIGITS, the solver's base_lr gets ignored entirely.

2. Caffe .prototxt's Learning Rate Params (lr_mult) #

• These are per-layer (or per-parameter) scaling factors defined in your network's prototxt, usually inside the param block of a layer. Here's a quick example:

  layer {
    name: "fc_final"
    type: "InnerProduct"
    # ... other layer config ...
    param { lr_mult: 1.0 decay_mult: 1.0 }  # For weights
    param { lr_mult: 2.0 decay_mult: 0.0 }  # For biases
  }
  

• Their job is to adjust the learning rate for specific parts of your network relative to the global base rate. For instance, you might want biases to learn faster than weights, or fine-tune a final fully connected layer with a higher rate than pre-trained conv layers—this is where lr_mult comes in.
• If you don't explicitly set lr_mult for a layer, Caffe defaults to a value of 1.0 for both weights and biases.

The real learning rate applied to each parameter in your network is a simple product of the two:
Actual Learning Rate = DIGITS Base Learning Rate × Corresponding Layer's lr_mult

Let's use a concrete example to make this tangible:

• Suppose you set DIGITS' Base Learning Rate to 0.001
• A conv layer with lr_mult: 1.0 uses 0.001 × 1.0 = 0.001
• The bias term of your final fully connected layer with lr_mult: 2.0 uses 0.001 × 2.0 = 0.002

That's the core of how these two settings interact—DIGITS gives you a global control knob, while lr_mult lets you tweak individual layers to suit your training needs.

内容的提问来源于stack exchange，提问作者chandan v