线性回归中梯度下降导致参数变为负无穷的问题排查

问题背景 #

尝试实现简单线性回归，编写了代价函数和梯度下降函数，代价函数运行正常，但梯度下降过程中参数突然跳变为离谱数值，最终出现inf和nan。

代价函数代码 #

m = len(x) # Training examples
def cost(w, b, x, y):
    j_wb = 0.
    for i in range(m):
        f_wb = w * x[i] + b
        err = f_wb - y[i]
        j_wb += err**2
    j_wb = j_wb / (m * 2)
    return j_wb

代价函数测试结果 #

at iteration 0: COST = 0.0
at iteration 1: COST = 11.906281776347848
at iteration 2: COST = 24.406303301163156
at iteration 3: COST = 25.89142657618535    
at iteration 4: COST = 31.71690104577324
at iteration 5: COST = 32.222444954452776
at iteration 6: COST = 52.79887560513525
at iteration 7: COST = 57.723294484239304
at iteration 8: COST = 59.252477506721256
at iteration 9: COST = 61.178601048944415
Final cost: 4.5292025547813575

梯度下降函数代码 #

def gradient(w, b, x, y, iterations, alphar):
    # Initialize
    dj_dw = 0
    dj_db = 0

    # Gradient descent
    for i in range(iterations):
        j_wb = cost(w, b, x, y)
        dj_dw = j_wb * x[i]
        dj_db = j_wb
        w = w - alphar * dj_dw
        b = b - alphar * dj_db
    return dj_dw, dj_db

问题表现 #

> Iteration 0 || Cost = 4.5292025547813575 || w = -0.08700861314752584
> Iteration 1 || Cost = 1919.5314293706836 || w = -959.8527232984893
> Iteration 2 || Cost = 1540639463.935084 || w = -231096879.44298592
> Iteration 3 || Cost = 8.924767986122691e+19 || w = -3.3914118347497325e+19
> Iteration 4 || Cost = 1.9197504655865147e+42 || w = -1.6701829050602676e+42
> Iteration 5 || Cost = 4.653919574489622e+87 || w = -1.675411046816264e+87
> Iteration 6 || Cost = 4.685387293902403e+177 || w = -5.622464752682884e+176
> Iteration 7 || Cost = inf || w = -inf
> Iteration 8 || Cost = nan || w = nan
> Iteration 9 || Cost = nan || w = nan
> Iteration 10 || Cost = nan || w = nan

错误原因分析 #

1. 梯度计算完全错误：梯度下降中dj_dw和dj_db的计算逻辑完全不符合线性回归的梯度定义。正确的梯度是代价函数对参数的偏导数：

   • 对w的偏导数：$\frac{\partial J(w,b)}{\partial w} = \frac{1}{m} \sum_{i=0}^{m-1} (f_{w,b}(x^{(i)}) - y^{(i)})x{(i)}$
   • 对b的偏导数：$\frac{\partial J(w,b)}{\partial b} = \frac{1}{m} \sum_{i=0}^{m-1} (f_{w,b}(x^{(i)}) - y^{(i)})$
     当前直接用代价函数值j_wb乘以x[i]和赋值给dj_db，完全偏离了梯度的正确计算方式，导致参数更新方向完全错误。

2. 迭代逻辑混淆：循环中的i是迭代次数，却被用来作为样本索引x[i]，即使迭代次数未超过样本数量，也只用到了单个样本的信息计算梯度，而非所有样本的累积，这也是梯度错误的核心原因之一。

3. 学习率放大错误影响：错误的梯度本身数值可能极大，再乘以学习率后，参数更新步长会变得离谱，直接导致代价函数指数级飙升，最终出现无穷大和NaN。

解决建议 #

1. 修正梯度计算逻辑 #

重新实现dj_dw和dj_db的计算，遍历所有样本累积误差项：

def gradient_descent(w, b, x, y, iterations, alpha):
    m = len(x)
    for iter_num in range(iterations):
        dj_dw = 0.
        dj_db = 0.
        # 遍历所有样本计算梯度
        for i in range(m):
            f_wb = w * x[i] + b
            err = f_wb - y[i]
            dj_dw += err * x[i]
            dj_db += err
        # 计算平均梯度
        dj_dw /= m
        dj_db /= m
        # 更新参数
        w = w - alpha * dj_dw
        b = b - alpha * dj_db
        # 打印迭代信息，验证代价变化
        current_cost = cost(w, b, x, y)
        print(f"Iteration {iter_num+1} || Cost = {current_cost} || w = {w} || b = {b}")
    return w, b

2. 调整学习率 #

梯度修正后，建议从较小的学习率开始尝试（比如0.001、0.01），观察代价函数是否逐步下降。如果代价函数震荡，继续减小学习率；如果下降过慢，可适当增大。

3. 验证迭代过程 #

每次迭代输出代价、w、b，确保代价函数是逐步降低的，这是梯度下降正常工作的核心标志。如果代价反而上升，说明学习率过大或梯度仍有错误。

内容的提问来源于stack exchange，提问作者Lucirie