Python实现线性回归：从零编码后损失上升问题排查

Hey there, let's figure out why your hand-built linear regression is seeing loss increase instead of converge—this is a super common pitfall when coding ML from scratch, so let's break down the most likely culprits and fixes.

Top Possible Issues & Fixes #

1. Gradient Calculation/Update Sign Error #

This is the #1 reason models diverge. Let's double-check the math:

• For SSE loss (sum((y_pred - y_true)**2)), the gradient for weights W is (2/N) * X.T @ (y_pred - y_true) (where N is sample count), and for bias B it's (2/N) * sum(y_pred - y_true).
• The critical part is how you apply this gradient: you need to subtract the gradient scaled by learning rate from your parameters. If you accidentally add it instead, you're pushing parameters away from the minimum, which causes loss to skyrocket.

Example of correct update logic:

# Compute predictions and error
y_pred = X @ W + B
error = y_pred - y_true

# Calculate gradients (the 2 can be absorbed into learning rate, so often omitted)
dW = (X.T @ error) / len(X)
dB = np.mean(error)

# Update parameters (note the minus sign!)
W = W - learning_rate * dW
B = B - learning_rate * dB

2. Learning Rate Is Too High #

Even if your gradients are correct, an overly large learning rate can cause your model to "overshoot" the loss minimum and diverge. Since you initialized W with the MLE closed-form solution (which is already the optimal value for linear regression without regularization), a big learning rate will immediately push W away from that optimal point, making loss jump up.

Try:

• Starting with an extremely small learning rate (e.g., 1e-4 or 1e-5)
• Gradually increasing it if loss converges too slowly
• Adding learning rate decay over epochs

3. Unstable MLE Initialization #

The closed-form solution W = (X.T @ X)^-1 @ X.T @ y relies on X.T @ X being invertible. If your dataset has multicollinearity (highly correlated features), X.T @ X becomes singular or nearly singular. This leads to unstable, large-magnitude weights that make the model prone to divergence during training.

Check for this by calculating the condition number of X.T @ X:

cond_number = np.linalg.cond(X.T @ X)
print(f"Condition number: {cond_number}")

A condition number > 1e10 indicates severe multicollinearity. Fixes include:

• Dropping redundant features
• Adding a small regularization term (ridge regression: W = (X.T @ X + λI)^-1 @ X.T @ y where λ is a small positive value like 1e-3)

4. Standardization Missteps #

You mentioned you standardized the data, but let's confirm a few details:

• Did you exclude the bias term from standardization? The bias B should not be scaled—standardize only your input features X.
• Did you fit the standardizer (compute mean/std) only on the training set, then apply it to validation/test sets? While this might not cause immediate divergence, it's a best practice that avoids data leakage.
• Are you using the correct standardization formula? It should be (X - mean) / std, not scaling to [0,1] (min-max scaling) unless that's intentional (though standardization is better for linear regression).

5. Loss Calculation Mistake #

Double-check that your loss function is computed correctly. For SSE, it should be np.sum((y_pred - y_true)**2) (order doesn't matter for the sum of squares, but consistency with gradient calculation is key). If you accidentally flipped y_true and y_pred in the gradient but not the loss, the direction of updates will still be wrong.

Quick Debugging Steps #

1. Manual Gradient Check: Pick 2-3 small samples, compute y_pred, error, and gradients by hand, then compare to your code's output. This will catch math errors fast.
2. Freeze Bias: Temporarily set dB = 0 and see if loss behaves differently—this can isolate whether the bias update is causing issues.
3. Plot Loss vs. Epochs: Visualizing the loss curve will tell you if it's diverging immediately (sign/learning rate issue) or slowly (unstable initialization).

内容的提问来源于stack exchange，提问作者VM_AI