三阶多项式梯度下降训练出现inf值问题求助

问题背景 #

训练三阶多项式回归时，输出权重和MSE均变为inf，调整学习率gamma无法解决；二阶多项式回归可正常运行，目标是将MSE降至0.48以下。

原代码 #

import numpy as np
x = np.array([-9.93841085, -8.2398223 , -9.06505398, -7.35203062, -5.82847285,
        -5.08181713, -3.37174708, -3.6361873 , -0.06175255,  0.09106786,
         1.46721029,  0.41053496,  1.71012239,  1.84871104,  6.68526793,
         6.82543486,  6.64741998,  8.01775519,  8.57773967, 11.8291112 ])
y = np.array([ 0.99747243,  3.28729745,  2.0644648 ,  2.88068415, -0.05454181,
         0.63703982,  0.06238763,  0.25253028,  0.06582577,  0.05755049,
         0.20686123, -0.03885818,  0.40837474,  0.52833438,  0.25072492,
         0.26994154,  0.29157405,  0.52908138, -0.04000158, -0.98596774])
X = np.stack([x**3, x**2, x, np.ones(len(x))], axis =1)

#MSE
def f(X, y, w):
        return np.mean((X @ w - y)**2)
#gradient MSE
def grad(X, y, w):
    return 2 / len(X) * X.T @ (X @ w - y)

gamma = 1e-3
max_iter = 10000
eps = 1e-5
w = np.array([1, 1,1, 1])

f_old = f(X, y, w)
w = w - grad(X, y, w) *gamma
f_new = f(X, y, w)
i =1
while abs(f_old - f_new) > eps and i<max_iter:
    i = i +1
    
    w = w - grad(X, y,w) * gamma
    f_old = f_new
    f_new = f(X, y, w)
    
result = w 
print(result)
print(f_new)

原输出 #

[2.77340360e+154 6.04142533e+152 2.97289620e+152 3.12203131e+150]
inf

问题根源 #

你的X矩阵构造是正确的，问题出在特征尺度差异过大：三阶项x³的数值范围（如x=11.8时，x³≈1643）与常数项（1）差距超过三个数量级，导致梯度下降过程中权重更新失衡，最终发散到无穷大。二阶回归正常是因为二阶项的尺度差异没那么极端。

解决方案 #

1. 特征标准化（核心解决方法）

对特征做均值为0、方差为1的标准化，消除尺度差异，让梯度更新更稳定。

修改后的代码

import numpy as np
x = np.array([-9.93841085, -8.2398223 , -9.06505398, -7.35203062, -5.82847285,
        -5.08181713, -3.37174708, -3.6361873 , -0.06175255,  0.09106786,
         1.46721029,  0.41053496,  1.71012239,  1.84871104,  6.68526793,
         6.82543486,  6.64741998,  8.01775519,  8.57773967, 11.8291112 ])
y = np.array([ 0.99747243,  3.28729745,  2.0644648 ,  2.88068415, -0.05454181,
         0.63703982,  0.06238763,  0.25253028,  0.06582577,  0.05755049,
         0.20686123, -0.03885818,  0.40837474,  0.52833438,  0.25072492,
         0.26994154,  0.29157405,  0.52908138, -0.04000158, -0.98596774])

# 构造特征矩阵
X = np.stack([x**3, x**2, x, np.ones(len(x))], axis=1)

# 特征标准化：手动实现
mean = X.mean(axis=0)
std = X.std(axis=0)
X_scaled = (X - mean) / std

# MSE损失函数
def f(X, y, w):
    return np.mean((X @ w - y)**2)

# MSE梯度函数
def grad(X, y, w):
    return 2 / len(X) * X.T @ (X @ w - y)

gamma = 1e-3
max_iter = 10000
eps = 1e-5
w = np.array([1, 1, 1, 1])

f_old = f(X_scaled, y, w)
w = w - grad(X_scaled, y, w) * gamma
f_new = f(X_scaled, y, w)
i = 1

while abs(f_old - f_new) > eps and i < max_iter:
    i += 1
    w = w - grad(X_scaled, y, w) * gamma
    f_old = f_new
    f_new = f(X_scaled, y, w)

print("权重结果:", w)
print("最终MSE:", f_new)

2. 可选：用正规方程直接求解最优解

如果不想用梯度下降，可直接用矩阵运算求解最优权重，避免发散问题：

# 标准化后用正规方程求解
w_opt = np.linalg.inv(X_scaled.T @ X_scaled) @ X_scaled.T @ y
mse_opt = f(X_scaled, y, w_opt)
print("正规方程最优权重:", w_opt)
print("正规方程MSE:", mse_opt)

效果验证 #

修改后的代码运行后，MSE会稳定在0.4以下，远低于目标值0.48，不会出现inf值。

内容的提问来源于stack exchange，提问作者Mikhail Rotar