感知器收敛证明中的学习率作用及实践取值咨询

Great question—this is such a common gap between textbook proofs and real-world implementation that a lot of folks stumble on it. Let’s break this down step by step.

Why do convergence proofs use learning rate = 1? #

The short answer: simplicity. The Novikoff convergence theorem (the formal proof for perceptron convergence) only requires the learning rate to be a positive constant—it doesn’t have to be 1. Setting it to 1 just cleans up the math: you don’t have to carry an extra coefficient through all the inequality steps, making the proof easier to follow and write. The core guarantee stays the same: as long as your data is linearly separable and you use a positive learning rate, the perceptron will eventually find a separating hyperplane.

How does learning rate affect convergence in practice? #

While the theorem says any positive lr works, the speed and stability of convergence depend a lot on its value:

• Too large a learning rate: Even though the theorem promises convergence, in practice you’ll see your weights bouncing wildly around the optimal solution. Imagine trying to walk to a target—if each step is 10 feet long, you’ll overshoot it every time, backtrack, and waste steps. With noisy data (which is almost always the case in real life), this can even lead to the model never settling on a good hyperplane.
• Too small a learning rate: Your model will take forever to converge. Each weight update is tiny, so you’ll need hundreds or thousands more iterations to reach the separating hyperplane. If you stop training early (which people often do), you might never get a good model at all.
• Just right: A well-chosen lr lets the weights move steadily toward the optimal solution—fast enough to make progress each iteration, but slow enough to avoid overshooting and oscillating.

Practical tips for choosing a learning rate #

Here’s what works in practice for perceptrons:

• Start with a small-to-moderate range: Try values like 0.01, 0.05, or 0.1 first. These are safe defaults that balance speed and stability for most linearly separable datasets.
• Watch your training curve: Plot the number of misclassified examples over iterations. If the error jumps up and down a lot, your lr is too big. If the error drops at a snail’s pace, go bigger.
• Scale your features first: If your features have wildly different scales (e.g., one feature is 0-1000 and another is 0-1), standardize them (normalize to 0-1 or z-score) before training. This makes your learning rate choices more consistent—you won’t have one feature dominating the weight updates just because it’s scaled larger.
• Decay the learning rate over time (optional): For even better results, you can start with a larger lr to make quick progress, then gradually reduce it (e.g., multiply by 0.9 every 100 iterations). This lets you avoid late-stage oscillations while still getting fast early convergence.

内容的提问来源于stack exchange，提问作者Qwerto