多项式回归梯度爆炸与欠拟合问题排查求助

问题背景 #

自行实现二次多项式回归模型（形式为$ax^2 + bx + c$），以$x^2$作为目标函数生成数据集，采用批量梯度下降训练50个epoch。训练中发现：

• 学习率$\geq0.001$时模型出现梯度爆炸
• 学习率$\leq0.0001$时模型出现欠拟合

梯度爆炸时，预测值虽数值偏大，但曲线形状符合正确多项式趋势；部分预测值为负时呈现镜像状态。

模型表现 #

梯度爆炸时的模型表现：

欠拟合时的模型表现：

代码实现 #

main.py #

from decimal import Decimal
from matplotlib.pyplot import plot, draw, pause, clf
from model import PolynomialRegression

POLYNOMIAL_FUNCTION = [0, 1, 2]
LEARNING_RATE = Decimal(0.0001)

DATASET = [0, 1, 2, 3, 4, 5, 6, 7]
LABELSET = [0, 1, 4, 9, 16, 32, 64, 128]

EPOCHS = 50

model = PolynomialRegression(POLYNOMIAL_FUNCTION, LEARNING_RATE)

for _ in range(EPOCHS):
    for data, label in zip(DATASET, LABELSET):
        # train the model
        model.train(data, label)

    # update the model
    model.update()

    # predict the dataset
    predictions = [model.predict(data) for data in DATASET]

    # plot predictions and labels
    plot(predictions)
    plot(LABELSET)

    draw()
    pause(0.1)
    clf()

    print(model.parameters)

    # erase the stored gradients
    model.clear_grad()

model.py #

from decimal import Decimal

class PolynomialRegression:
    """
    Polynomial regression model.
    """

    def __init__(self, polynomial_function: list, learning_rate: Decimal) -> None:
        # the structure of the polynomial function (the exponents)
        self.polynomial_function = polynomial_function

        # parameters of the model set to be 1
        self.parameters = [Decimal(1)] * len(polynomial_function)

        self.learning_rate = learning_rate

        # stored gradients to update the model
        self.gradients = []
    
    def predict(self, x: Decimal) -> Decimal:
        """
        Make a prediction based on the input.

        Args:
            x (Decimal): Input to the model.

        Returns:
            Decimal: A prediction.
        """

        y = Decimal(0)

        # go through each parameter and exponent
        for param, exponent in zip(self.parameters, self.polynomial_function):
            # compute a term and add it to the final output
            y += param * (x ** exponent)
        
        return y
    
    def train(self, x: Decimal, y: Decimal) -> Decimal:
        """
        Compute a gradient from a given input and target output.

        Args:
            x (Decimal): Input for the model.
            y (Decimal): Target/Desired output.
        
        Returns:
            Decimal: An MSE loss.
        """

        prediction = self.predict(x)

        error = prediction - y
        loss = error ** 2

        gradient = []

        # go through each parameter and exponent
        for param, exponent in zip(self.parameters, self.polynomial_function):
            # compute the gradient for a single parameter
            param_gradient = error * (x ** exponent) * self.learning_rate
            
            # add the parameter gradient to the gradient list
            gradient.append(param_gradient)

        # add the gradient to a list
        self.gradients.append(gradient)

        return loss
    

    def __sum_gradients(self) -> list[Decimal]:
        """
        Return a sum of gradients along the 0 axis.
        (equivalent of numpy.sum(x, axis=0))

        Returns:
            list: List of summed Decimals.
        """

        result = [Decimal(0)] * len(self.parameters)

        # iterate through the y axis
        for gradient in self.gradients:
            # iterate through the x axis
            for i, param_gradient in enumerate(gradient):
                result[i] += param_gradient
        
        return result

    def update(self) -> None:
        """
        Update the model's parameters based on the stored gradients.
        """

        summed_gradients = self.__sum_gradients()

        # fraction used to calculate the average for every gradient
        averaging_fraction = Decimal(1) / len(self.gradients)

        for param_index, grad in enumerate(summed_gradients):
            self.parameters[param_index] -= averaging_fraction * grad
        
    def clear_grad(self) -> None:
        """
        Clear/Reset the stored gradients.
        """

        self.gradients = []

问题排查与修复建议 #

1. 标签数据集错误 #

核心问题：LABELSET中x=5、6、7对应的标签是32、64、128，这是$2^{x$的结果，而非目标函数$x}2$的正确值（25、36、49）。错误的标签会让模型学习完全偏离的目标，直接导致训练异常。

修复：修正标签集为正确的$x^2$值：

LABELSET = [x**2 for x in DATASET]  # 结果为[0,1,4,9,16,25,36,49]

2. 梯度计算缺失系数 #

MSE损失函数$Loss=(y_{pred}-y_{true})^{2$对参数$\theta_i$的梯度应为$2*(y_{pred}-y_{true})*x}{e_i}$，代码中param_gradient缺少了系数2，导致梯度幅度减半，收敛速度变慢，需要更大的学习率才能达到收敛效果，也是欠拟合的诱因之一。

修复：在梯度计算中添加系数2：

param_gradient = 2 * error * (x ** exponent) * self.learning_rate

3. 输入数据未归一化 #

输入x的取值范围是0-7，$x^2$的最大值为49，较大的数值会导致梯度随x增大而急剧变大，容易引发梯度爆炸。对输入数据做归一化可以缩小梯度幅度，稳定训练过程。

修复：将x缩放到0-1区间：

# 在main.py中，将DATASET替换为归一化后的数据
MAX_X = max(DATASET)
DATASET_NORMALIZED = [Decimal(x)/MAX_X for x in DATASET]
# 后续训练和预测使用DATASET_NORMALIZED

4. 参数初始化优化 #

初始参数全设为1，对于二次项来说初始预测值为$x^2+x+1$，与真实值偏差较大。可以改为初始化接近0的随机值或直接设为0，加快收敛速度。

修复：修改参数初始化逻辑：

# 替换__init__中的参数初始化
from random import uniform
self.parameters = [Decimal(uniform(-0.1, 0.1)) for _ in polynomial_function]
# 或者直接初始化为0
# self.parameters = [Decimal(0)] * len(polynomial_function)

修复后训练效果 #

修正上述问题后，使用学习率0.01即可稳定收敛，模型预测值会贴合$x^2$曲线，不会出现梯度爆炸或欠拟合情况。

内容的提问来源于stack exchange，提问作者Vladislav Korecký