线性回归（波士顿房价模型）IndexError问题排查求助

我尝试基于波士顿房价数据集训练线性回归模型，由于新版Scikit-learn移除了该内置数据集，已通过fetch_openml获取数据（代码第7、8行）。运行时触发索引越界错误，怀疑是自定义permute_data函数的索引排列问题，将X[perm]改为X[perm-1]后问题仍存在。

完整代码 #

import numpy as np
from numpy import ndarray
from typing import Tuple
from typing import Dict
import matplotlib.pyplot as plt
import pandas
from sklearn.datasets import fetch_openml 
boston = fetch_openml(name='boston', version=1) 
data = boston['data']
target = boston['target']
features = boston['feature_names']
from sklearn.preprocessing import StandardScaler
s = StandardScaler()
data = s.fit_transform(data)
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test = train_test_split(data,target,test_size=0.30,random_state= 80718)
y_train = np.reshape(-1,1)
y_test = np.reshape(-1,1)
def init_weights(rows):
    W = np.random.randn(rows,1)
    B = np.random.randn(1,1) # Try with B as only integer
    weights = {}
    weights['W'] = W
    weights['B'] = B
    return weights
def linear_forward(X_batch:ndarray,y_batch:ndarray,weights: Dict[str,ndarray]) -> Tuple[float,dict[str,ndarray]]:
    d  = np.dot(X_batch,weights['W'])
    P :ndarray = d + weights['B']
    loss = np.mean(np.power(y_batch - P,2))
    forward_info :Dict[str,ndarray]= {}
    forward_info['X'] = X_batch
    forward_info['N'] = d
    forward_info['P'] = P
    forward_info['y'] = y_batch
    return forward_info, loss
def loss_gradients(forward_info:Dict[str,ndarray],weights:Dict[str,ndarray]):
    dLdP = -1*2*(forward_info['y'] - forward_info['P'])
    dPdW = np.transpose(forward_info['X'])
    dLdW = np.dot(dPdW,dLdP)
    dPdB = np.ones_like(weights['B'])
    dLdB = (dLdP * dPdB).sum(axis = 0)
    loss_gradients :Dict[str,ndarray]= {}
    loss_gradients['W'] = dLdW # Change of loss with respect to our functions 
    loss_gradients['B'] = dLdB 
    return loss_gradients
def generate_batch(X:ndarray,y:ndarray,start,batch_size): # Yekda nusta return last X[last] karun baghu
    assert X.ndim == y.ndim == 2
    if start + batch_size > X.shape[0]:
        batch_size = X.shape[0] - start
    X_batch,y_batch = X[start: start+batch_size],y[start:start+batch_size]
    return X_batch,y_batch 
def permute_data(X:ndarray,y:ndarray):
    perm = np.random.permutation(X.shape[0])
    return X[perm],y[perm]
def train(X:ndarray,y:ndarray,n_iter,batch_size,learning_rate,return_weights:bool,seed):
    if seed:                    # ......................nahi kalaala
        np.random.seed(seed)
    start = 0 # Nahi kalaala
    weights = init_weights(X.shape[1])
    X,y = permute_data(X,y)
    losses = []
    for i in range(n_iter):
        if start >= X.shape[0]:
            X,y = permute_data(X,y)
            start = 0
        X_batch,y_batch = generate_batch(X,y,start,batch_size)
        start+=batch_size # start = start + batch_size
        forward_info,loss = linear_forward(X_batch,y_batch,weights)
        losses.append(loss)
        loss_grads = loss_gradients(forward_info,weights)
        for key in weights.keys():
            weights[key] -= learning_rate*loss_grads[key]
    if return_weights:
        return losses,weights
train_info = train(X_train,y_train,1000,23,0.001,True,80718) #Seeds Badlun Baghu
losses = train_info[0]
weights_trained = train_info[1]
plt.plot(list(range(1000)), losses)
def predict(X: ndarray,weights:Dict[str,ndarray]): #weights to be put here are the ones obtained from training
    N = np.dot(X,weights['W'])
    P = N + weights['B']
    return P
preds = predict(X_test,weights_trained)
def mae(preds: ndarray, actuals: ndarray):
    return np.mean(np.abs(preds - actuals))
def rmsE(preds:ndarray,actuals:ndarray):
    return np.sqrt(np.mean(np.power(actuals - preds,2)))
print("Mean absolute error:", round(mae(preds, y_test), 4), 
      "Root mean squared error:", round(rmsE(preds, y_test), 4))

报错信息 #

Traceback (most recent call last):
  File "Documents/Code/Deep Learning/hu.py", line 75, in <module>
    train_info = train(X_train,y_train,1000,23,0.001,True,80718) #Seeds Badlun Baghu
  File "Documents/Code/Deep Learning/hu.py", line 60, in train
    X,y = permute_data(X,y)
  File "Documents/Code/Deep Learning/hu.py", line 54, in permute_data
    return X[perm],y[perm]
IndexError: index 188 is out of bounds for axis 0 with size 1

问题根源 #

报错提示index 188 is out of bounds for axis 0 with size 1，说明y_train被错误修改为仅1行的数组，而permute_data生成的排列索引最大对应原训练集的样本数（约354），远大于y的行数，触发索引越界。

问题出在reshape的调用方式：

y_train = np.reshape(-1,1)
y_test = np.reshape(-1,1)

该写法未传入y_train作为参数，而是直接将np.reshape(-1,1)的结果赋值给y_train，返回的是形状为(1,1)的数组，完全破坏了原目标变量的结构。

修复步骤 #

1. 修正reshape代码：
   由于fetch_openml返回的target是pandas Series类型，需先转为numpy数组再调整维度：

   y_train = np.asarray(y_train).reshape(-1, 1)
   y_test = np.asarray(y_test).reshape(-1, 1)
   

   也可以用Series的values属性直接转数组：

   y_train = y_train.values.reshape(-1, 1)
   y_test = y_test.values.reshape(-1, 1)
   

2. 验证维度：修改后执行print(y_train.shape)，应输出(354, 1)（波士顿数据集共506条样本，70%作为训练集），而非(1,1)。

3. 保留原permute_data函数：该函数逻辑正确，无需修改，排列索引会匹配X和y的样本数。

额外注意事项 #

• fetch_openml返回的特征和目标变量可能是pandas对象，使用前转为numpy数组可避免维度适配问题。
• 训练时batch_size设为23是合理的，训练集样本数354可被23整除15次，最后一批剩余9个样本会被generate_batch自动处理。

内容的提问来源于stack exchange，提问作者Soham Ghodake