基于MSE的二次函数拟合代码故障排查：梯度下降法失效问题

Let’s dig into why your quadratic fitting code isn’t performing as expected, especially that lagging c parameter. You’re on the right track with gradient descent (borrowing from neural nets/robotics Jacobians), but there are a few critical issues throwing things off:

1. Learning Rate is Far Too Small & Misaligned with Gradient Scales #

Your learning rate lr = 0.0000000001 is tiny—way too tiny to make meaningful progress for the c parameter. Here’s why:

• The gradient for a (deda) scales with x², which gets large as x increases (e.g., if your dataset has 100 points, x² hits 9801).
• The gradient for c (dedc) is a constant scale (no x term), so its magnitude is way smaller than deda or dedb.
  With the same tiny learning rate, a and b might get small updates, but c barely moves at all—hence the "lag" you’re seeing.

2. Momentum Update Logic is Buggy #

Your momentum calculation is double-counting the correction term, which messes up parameter updates:

a = a - correction - a_momentum

Since a_momentum = 0.99*a_momentum + 0.1*correction, you’re subtracting correction plus a term that already includes 10% of correction. This means you’re effectively subtracting ~1.1x the correction plus a decaying old momentum, leading to unstable or overly small updates (worse for c, which already has a tiny gradient).

The standard momentum approach is to apply the momentum term as the update itself, like:

a_momentum = 0.9 * a_momentum + correction  # Adjust beta (0.9 is common)
a = a - a_momentum

This way, momentum accumulates past updates without double-dipping on the current correction.

3. No Feature Scaling (Gradient Magnitude Imbalance) #

Quadratic terms (x²) naturally have much larger magnitudes than linear (x) or constant terms. This creates a huge imbalance in gradient sizes between a, b, and c. Even with a better learning rate, c will always update slower because its gradient is orders of magnitude smaller.

Fix this by scaling your input x values (e.g., normalize to the range [0,1] or standardize to mean 0, variance 1). This evens out the gradient scales across all parameters, making gradient descent work uniformly for a, b, and c.

Fixed Code Example #

Here’s a revised version addressing these issues:

import math as m

def MSE(y_true, y_pred):
    return sum((y_t - y_p)**2 for y_t, y_p in zip(y_true, y_pred)) / len(y_true)

def quadraticRegression(data, dErr):
    # Normalize x values to [0,1] to balance gradients
    x_vals = [x / (len(data)-1) for x in range(len(data))]
    
    a = 1.0
    b = 1.0
    c = 1.0
    a_momentum = 0.0
    b_momentum = 0.0
    c_momentum = 0.0
    
    # Initial estimate with normalized x
    estimate = [a*x**2 + b*x + c for x in x_vals]
    error = MSE(data, estimate)
    errorOld = error
    lr = 0.1  # Much larger, feasible with scaled x
    
    while abs(error - errorOld) > dErr:
        # Gradient calculations with normalized x
        deda = sum(2 * x**2 * (a*x**2 + b*x + c - data[i]) for i, x in enumerate(x_vals)) / len(data)
        dedb = sum(2 * x * (a*x**2 + b*x + c - data[i]) for i, x in enumerate(x_vals)) / len(data)
        dedc = sum(2 * (a*x**2 + b*x + c - data[i]) for i, x in enumerate(x_vals)) / len(data)
        
        # Corrected momentum updates
        correction_a = deda * lr
        a_momentum = 0.9 * a_momentum + correction_a
        a -= a_momentum
        
        correction_b = dedb * lr
        b_momentum = 0.9 * b_momentum + correction_b
        b -= b_momentum
        
        correction_c = dedc * lr
        c_momentum = 0.9 * c_momentum + correction_c
        c -= c_momentum
        
        # Update error tracking
        estimate = [a*x**2 + b*x + c for x in x_vals]
        errorOld = error
        error = MSE(data, estimate)
        # print(f"Current error: {error:.6f}")  # Optional debug print
    
    return a, b, c, error

Key Takeaways #

• Tune learning rate relative to gradient scales: Use feature scaling to make this easier, or switch to adaptive optimizers (like Adam) that adjust learning rates per parameter automatically.
• Fix momentum logic: Don’t double-subtract correction terms—let momentum accumulate past updates smoothly.
• Validate gradient magnitudes: Print deda, dedb, dedc during training to confirm they’re in a similar range (feature scaling helps here).

内容的提问来源于stack exchange，提问作者Nyxeria