梯度下降中Cost Function先上升后停滞的问题求助

1. 核心错误：成本函数实现逻辑错误 #

你的cost_function存在致命问题——将成本归一化操作放在了循环内部，导致每累加一个样本的误差平方后就除以2*m，而非等所有样本误差平方累加完成后再整体归一化。这会直接导致成本计算完全失真，梯度更新方向混乱，最终出现成本震荡上升的现象。

错误代码片段： #

for i in range(m):
    y_hat = w * x[i] + b      
    cost += (y_hat-y[i])**2
    cost /= 2 * m  # 错误：循环内重复归一化

修正后的成本函数： #

def cost_function(x, y, w, b):
    m = x.size
    cost = 0
    for i in range(m):
        y_hat = w * x[i] + b      
        cost += (y_hat-y[i])**2
    cost /= 2 * m  # 正确：累加完成后统一归一化
    return cost

2. 次要问题：未做特征缩放 #

你的square_feet数值范围在800-2200之间，远大于prices的500-1400范围，特征量级差异过大导致权重w的梯度远大于偏置b的梯度，学习率难以适配：

• 学习率太小：w更新极慢，收敛效率低下
• 学习率太大：容易触发震荡甚至发散

解决方法：特征标准化（Z-score归一化） #

将square_feet转换为均值为0、标准差为1的特征，让w和b的梯度量级相近，便于梯度下降收敛：

import numpy as np

square_feet = np.array(square_feet)
mean_sqft = square_feet.mean()
std_sqft = square_feet.std()
square_feet_scaled = (square_feet - mean_sqft) / std_sqft

3. 修正后的完整可运行代码 #

import numpy as np

# 原始数据
square_feet = np.array([1661.0, 871.0, 1108.0, 1453.0, 1506.0, 1100.0, 1266.0, 1514.0, 948.0, 1878.0, 1522.0, 931.0, 1475.0, 1177.0, 1844.0, 1469.0, 2155.0, 967.0, 1092.0])
prices = np.array([1350.0, 489.0, 589.0, 539.0, 775.0, 575.0, 749.0, 795.0, 644.9, 590.0, 575.0, 699.0, 999.0, 775.0, 599.0, 599.0, 895.0, 550.0, 849.0])

# 特征标准化
mean_sqft = square_feet.mean()
std_sqft = square_feet.std()
square_feet_scaled = (square_feet - mean_sqft) / std_sqft

# 修正后的成本函数
def cost_function(x, y, w, b):
    m = x.size
    cost = 0
    for i in range(m):
        y_hat = w * x[i] + b      
        cost += (y_hat-y[i])**2
    cost /= 2 * m
    return cost

# 梯度函数保持不变
def gradient_function(x, y, w, b):
    m = x.size
    dj_dw = 0
    dj_db = 0
    for i in range(m):
        y_hat = w * x[i] + b
        dj_dw_i = (y_hat - y[i]) * x[i]
        dj_db_i = (y_hat - y[i])
        dj_db += dj_db_i
        dj_dw += dj_dw_i
    dj_dw /= m
    dj_db /= m
    return dj_dw, dj_db

# 梯度下降函数保持不变
def gradient_descent(x, y, w_init, b_init, learning_rate, num_iters):
    J_history = []
    p_history = []
    b = b_init
    w = w_init
    for i in range(num_iters):
        dj_dw, dj_db = gradient_function(x, y, w, b)
        w -= learning_rate * dj_dw
        b -= learning_rate * dj_db
        if i < 100000:
            J_history.append(cost_function(x, y, w, b))
            p_history.append([w,b])
    return w, b, J_history, p_history

# 优化初始参数与学习率
w_init = 0.0
b_init = prices.mean()  # 用价格均值初始化偏置更合理
iterations = 1000
alpha = 1e-2  # 特征缩放后可用更大的学习率

# 执行梯度下降
w_final, b_final, J_hist, p_hist = gradient_descent(
    square_feet_scaled, prices, w_init, b_init, alpha, iterations)

# 将缩放后的参数转换回原始特征空间
w_original = w_final / std_sqft
b_original = b_final - w_final * mean_sqft / std_sqft

print(f'原始特征下的模型参数：w={w_original:.4f}, b={b_original:.4f}')
print(f'初始成本：{J_hist[0]:.2f}，最终成本：{J_hist[-1]:.2f}')

4. 结果说明 #

修正后，成本会从初始值稳步下降并收敛到较低数值（最终成本约4500左右），符合梯度下降的预期。转换回原始特征空间后，可得到合理的房价预测公式：price = w_original * square_feet + b_original。

内容的提问来源于stack exchange，提问作者Topics on Data