R语言梯度下降代码异常求助：参数估计线性递增问题

Hey there! Let's figure out why your gradient descent is spitting out linearly increasing parameter estimates instead of converging to the right values for the cars dataset. This is a super common issue, and usually boils down to a few key mistakes—let's break them down and fix your code.

Common Culprits & Fixes #

1. You're Updating Parameters in the Wrong Direction #

The biggest red flag for exploding parameters is updating parameters with the gradient instead of against it. Gradient descent works by moving opposite to the direction of the loss function's slope (the gradient).

If your code looks like this (wrong):

b0 <- b0 + alpha * gradient_b0
b1 <- b1 + alpha * gradient_b1

That's why your parameters are blowing up. You need to subtract the gradient times the learning rate:

b0 <- b0 - alpha * gradient_b0
b1 <- b1 - alpha * gradient_b1

2. No Feature Scaling (Critical for cars!) #

The cars dataset has speed (range 4–25) and dist (range 2–120)—huge differences in scale. Without scaling, the gradient for the slope (b1) will be way larger than the gradient for the intercept (b0), making your updates unstable. Even a "reasonable" learning rate will cause b1 to jump around or explode.

Fix this by standardizing your features first:

# Scale both variables to mean=0, sd=1
cars_scaled <- cars
cars_scaled$speed <- scale(cars_scaled$speed)
cars_scaled$dist <- scale(cars_scaled$dist)

3. Incorrect Gradient Calculation #

Double-check your gradient formulas for linear regression (using MSE loss):

• Gradient for intercept (b0): (1/n) * sum(y_hat - y)
• Gradient for slope (b1): (1/n) * sum((y_hat - y) * x)

If you flipped the error term to y - y_hat, your gradient sign will be wrong—leading to updates in the wrong direction (and divergence).

4. Learning Rate (alpha) Is Too Large #

Even with scaling, an alpha that's too big can cause the algorithm to overshoot the minimum and diverge. Start small (like 0.01) and adjust up if convergence is too slow, or down if parameters explode.

Fixed Full Code Example #

Here's a complete, working version of your function that collects parameter estimates and plots the search path:

GradientDescent <- function(b0_start, b1_start, x, y, niter=100, alpha=0.01) {
  # Initialize data frame to track parameters
  params_df <- data.frame(iter = 1:niter, b0 = numeric(niter), b1 = numeric(niter))
  b0 <- b0_start
  b1 <- b1_start
  n <- length(y)
  
  for (i in 1:niter) {
    # Calculate predictions
    y_hat <- b0 + b1 * x
    # Compute gradients (correct sign and formula)
    gradient_b0 <- (1/n) * sum(y_hat - y)
    gradient_b1 <- (1/n) * sum((y_hat - y) * x)
    # Update parameters (negative gradient direction)
    b0 <- b0 - alpha * gradient_b0
    b1 <- b1 - alpha * gradient_b1
    # Save current parameters
    params_df$b0[i] <- b0
    params_df$b1[i] <- b1
  }
  return(params_df)
}

# Use scaled cars data
cars_scaled <- cars
cars_scaled$speed <- scale(cars_scaled$speed)
cars_scaled$dist <- scale(cars_scaled$dist)

# Run gradient descent
params <- GradientDescent(b0_start = 0, b1_start = 0, 
                          x = cars_scaled$speed, y = cars_scaled$dist,
                          niter = 200, alpha = 0.1)

# Check parameter progression
head(params)

# Plot the search path in parameter space
plot(params$b1, params$b0, type = "l", 
     xlab = "Slope (b1)", ylab = "Intercept (b0)", 
     main = "Gradient Descent Search Path")
# Add the true OLS estimate (red dot)
true_coeffs <- lm(dist ~ speed, data = cars_scaled)$coefficients
points(true_coeffs[2], true_coeffs[1], col = "red", pch = 19, cex = 1.5)

Quick Checks for Your Original Code #

1. Verify you're subtracting (not adding) the gradient during parameter updates
2. Check if you scaled your features before running the algorithm
3. Confirm your gradient calculations match the formulas above
4. Try reducing your learning rate if you still see divergence

内容的提问来源于stack exchange，提问作者invictus