单变量线性回归出现水平斜率与高成本值问题求助

问题描述 #

手动实现单变量线性回归模型时遇到问题：预测结果始终为水平直线，成本值卡在908787无下降。已尝试调整学习率和初始theta值，问题依旧。

原始代码 #

import numpy as np
from sklearn.datasets import load_diabetes
import matplotlib.pyplot as plt

# Dataset Preparation
diabetes = load_diabetes()
X = diabetes.data
Y = diabetes.target

# Use only one feature (BMI)
X = X[:, np.newaxis, 2]
X = X.reshape((-1,1))

Y = np.expand_dims(Y, 1)
data = np.append(X, Y, 1)

# random shuffle
np.random.seed(1201)
np.random.shuffle(data)

# split data
total_sample = len(data)
train = data[:int(total_sample*0.70)]
dev = data[int(total_sample*0.70):int(total_sample*0.85)]
test = data[int(total_sample*0.85):]

def get_features_and_labels(data):
  features = data[:,:-1]
  labels = data[:,-1]
  return features, labels

train_x, train_y = get_features_and_labels(train)
dev_x, dev_y = get_features_and_labels(dev)
test_x, test_y = get_features_and_labels(test)

# Linear Regression Model

def univariate_linear_regression(theta, input):
  pred = theta[0] + theta[1]*input
  return pred

# Cost Function
def compute_cost(Y_pred, Y_true):
  J = 1/(2*m) * np.sum((Y_pred - Y_true)**2) # Mean Square Error
  return J

def update_theta(theta, X, Y_true, Y_pred, lr):
  theta[0] = theta[0] - (lr * (1/m) * np.sum(Y_pred - Y_true))
  theta[1] = theta[1] - (lr * (1/m) * np.sum((Y_pred - Y_true) * X))
  return theta

theta = [0.0, 0.0]
lr = 0.0001
m = len(train_y)
k = 0
plt.figure(figsize=(30, 30))

for i in range(500):
  pred = univariate_linear_regression(theta, train_x)
  cost = compute_cost(pred, train_y)
  theta = update_theta(theta, train_x, train_y, pred, lr)


  if(i%20==0):
    print(f"Iteration {i}, Cost: {cost}, Theta: {theta}")
    k +=1
    plt.subplot(5, 5, k)
    plt.scatter(train_x, train_y, color='b')
    plt.plot(train_x, pred, 'g')

s = 'theta:[%.4f, %.4f]' %(theta[0], theta[1])
    c = 'cost: %.4f' %cost
    plt.title(s+'\n'+c)

训练可视化结果 #

错误排查与修正方案 #

1. 学习率过小导致参数更新停滞 #

diabetes数据集的BMI特征已标准化到[-1,1]区间，但你设置的学习率lr=0.0001太小，使得theta1的更新步长微乎其微，始终接近初始值0，因此预测线保持水平。

修正： 将学习率调整为0.1或1，大幅提升参数更新速度。

2. 成本函数依赖全局变量存在风险 #

compute_cost函数中直接使用全局变量m，虽然当前场景未出错，但代码耦合性高，建议改为从输入的标签数据中获取样本数：

def compute_cost(Y_pred, Y_true):
    m = len(Y_true)
    J = 1/(2*m) * np.sum((Y_pred - Y_true)**2)
    return J

3. 使用列表存储theta效率低且易出问题 #

用列表存储theta会导致每次更新时的类型转换开销，建议改为numpy数组：

theta = np.array([0.0, 0.0])

修正后的训练循环示例 #

theta = np.array([0.0, 0.0])
lr = 0.1  # 调整学习率
m = len(train_y)
k = 0
plt.figure(figsize=(15, 15))

for i in range(500):
    pred = univariate_linear_regression(theta, train_x)
    cost = compute_cost(pred, train_y)
    theta = update_theta(theta, train_x, train_y, pred, lr)

    if(i%20==0):
        print(f"Iteration {i}, Cost: {cost:.2f}, Theta: [{theta[0]:.2f}, {theta[1]:.2f}]")
        k +=1
        plt.subplot(5, 5, k)
        plt.scatter(train_x, train_y, color='b')
        plt.plot(train_x, pred, 'g')
        s = 'theta:[%.4f, %.4f]' %(theta[0], theta[1])
        c = 'cost: %.4f' %cost
        plt.title(s+'\n'+c)

plt.tight_layout()
plt.show()

调整后，theta1会快速更新，预测线将逐渐拟合数据分布，成本值也会持续下降。

内容的提问来源于stack exchange，提问作者Ami