小批量/随机梯度下降是否隐含模拟退火相似效果？及sharp minima含义求证

1. Do mini-batch/stochastic gradient descent (SGD) have effects similar to simulated annealing? #

Great observation—there are indeed some overlapping intuitions between the two, but they operate differently under the hood:

• Shared core idea: Randomness for exploration
  Both methods leverage randomness to avoid getting trapped in poor local minima. Simulated annealing uses "temperature" to control the probability of accepting worse solutions (with randomness decreasing as training progresses), while mini-batch/SGD gets its randomness from sampling small subsets of data, which introduces noise into the gradient estimate. In early training, both have high randomness (high temperature for SA, noisy gradients for SGD) to explore the loss surface broadly; as training continues, randomness diminishes (temperature cools, learning rate decays or batch size grows) to converge to a stable minimum.

• Key differences

  • Simulated annealing explicitly accepts worse solutions via the Metropolis criterion—a deliberate mechanism to escape local minima.
  • SGD’s ability to escape is indirect: gradient noise from mini-batches can nudge the model out of sharp local minima basins, but it doesn’t intentionally move to higher-loss regions. The noise is a byproduct of approximate gradient calculation, not a designed acceptance rule.

In short: While they aren’t identical, mini-batch/SGD does share the implicit effect of simulated annealing—using controlled randomness to improve global search and avoid suboptimal local minima.

2. Is my understanding of "sharp minima" correct? #

Your intuition is spot-on! Let’s formalize it using the paper you referenced:

The paper defines sharp minima as regions of the loss surface where the minimum is surrounded by steep, rapid changes in loss (i.e., large gradients in nearby areas). Mathematically, this corresponds to the Hessian matrix at the minimum having large positive eigenvalues—meaning the curvature of the loss surface is very high around that point.

In contrast, flat minima are regions where the loss surface is gentle around the minimum (small Hessian eigenvalues), so small perturbations to model parameters don’t cause large jumps in loss.

The paper’s key point is that large-batch training, which produces more accurate (low-noise) gradient estimates, tends to converge to these sharp minima because precise gradients guide the model directly into narrow, steep basins. Mini-batch/SGD, however, uses gradient noise to "jiggle" the model out of these sharp basins, allowing it to find flatter minima that generalize better to unseen data—since flat minima are more robust to parameter changes and dataset variations.

内容的提问来源于stack exchange，提问作者Make42