采用2D Linear Regression填充缺失值输出无意义，求问题排查

问题背景与重现 #

手头有数据集，尝试通过2D回归拟合周边曲线斜率来近似填充缺失值，但输出结果不符合预期。以下是示例数据与代码：

示例数据集 #

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

local_window = pd.DataFrame({102.5: {0.021917: 0.0007808776581961896,
                                 0.030136: 0.0009108521507099643,
                                 0.035616: 0.001109650616093018,
                                 0.041095: 0.0013238862647034224,
                                 0.060273: 0.0018552410055933753},
                         105.0: {0.021917: 0.0008955896980595855,
                                 0.030136: 0.001003244315807649,
                                 0.035616: 0.0011852612740301449,
                                 0.041095: 0.0013952857530607904,
                                 0.060273: 0.0018525880756980716},
                         107.5: {0.021917: np.nan,
                                 0.030136: 0.0012354997955153118,
                                 0.035616: 0.00140044893559622,
                                 0.041095: 0.0015902024099268574,
                                 0.060273: 0.001973254493672934}})

现有填充函数 #

def predict_nan_local(local_window):
    if not local_window.isnull().values.any():
        return local_window
    
    # Extract x and y values for the local window
    X_local = local_window.columns.values.copy()
    y_local = local_window.index.values.copy()

    # Create a meshgrid of x and y values
    X_local, y_local = np.meshgrid(X_local, y_local)

    # Flatten x and y for fitting the model
    X_local_flat = X_local.flatten()
    y_local_flat = y_local.flatten()
    values_local_flat = local_window.values.flatten()
    
    # Find indices of non-NaN values
    non_nan_indices = ~np.isnan(values_local_flat)

    # Filter out NaN values
    X_local_flat_filtered = X_local_flat[non_nan_indices]
    y_local_flat_filtered = y_local_flat[non_nan_indices]
    values_local_flat_filtered = values_local_flat[non_nan_indices]

    regressor = LinearRegression()
    regressor.fit(np.column_stack((X_local_flat_filtered, y_local_flat_filtered)), values_local_flat_filtered)
    
    nan_indices = np.argwhere(np.isnan(local_window.values))
    X_nan = local_window.columns.values[nan_indices[:, 1]]
    y_nan = local_window.index.values[nan_indices[:, 0]]
    
    # Predict missing value
    predicted_values = regressor.predict(np.column_stack((X_nan, y_nan)))

    local_window.iloc[nan_indices[:, 0], nan_indices[:, 1]] = predicted_values

    return local_window

数据可视化结果 #

问题根源 #

1. 全局线性回归假设不成立：现有代码用普通线性回归拟合整个窗口的2D平面，但从可视化结果看，数据的变化趋势是非线性的（随着x、y增大，值的增长速率有变化），全局线性模型无法捕捉这种局部趋势，导致预测值偏差。
2. 忽略局部特性：你原本想利用"周边曲线斜率"填充，但代码用所有非NaN点拟合全局模型，没有聚焦缺失值附近的局部数据，无法反映周边的真实斜率变化。
3. 特征尺度差异：x轴（列名102.5等）与y轴（索引0.0219等）数值尺度差距极大，线性回归会自动偏向尺度更大的特征，进一步扭曲拟合结果。

改进思路与实现 #

1. 二维插值（推荐） #

利用scipy的网格插值，针对缺失值的邻近点做局部拟合，支持线性、立方等多种方法，更贴合数据的非线性趋势：

from scipy.interpolate import griddata

def fill_nan_with_interpolation(local_window):
    # 提取网格坐标与对应值
    x_cols = local_window.columns.values
    y_rows = local_window.index.values
    X_grid, Y_grid = np.meshgrid(x_cols, y_rows)
    value_grid = local_window.values

    # 整理非NaN点的坐标与值
    valid_points = np.column_stack((X_grid[~np.isnan(value_grid)], Y_grid[~np.isnan(value_grid)]))
    valid_values = value_grid[~np.isnan(value_grid)]

    # 获取缺失点坐标并插值
    nan_points = np.column_stack((X_grid[np.isnan(value_grid)], Y_grid[np.isnan(value_grid)]))
    filled_values = griddata(valid_points, valid_values, nan_points, method='cubic')

    # 填充回原DataFrame
    local_window.values[np.isnan(value_grid)] = filled_values
    return local_window

2. 局部单变量回归 #

针对缺失值所在的行/列，用邻近的非NaN点做单变量回归，贴合你想利用"周边曲线斜率"的需求：

def fill_nan_local_regression(local_window):
    nan_indices = np.argwhere(np.isnan(local_window.values))
    for row_idx, col_idx in nan_indices:
        # 优先用同一行的邻近数据拟合
        row_data = local_window.iloc[row_idx].dropna()
        if len(row_data) >= 2:
            reg = LinearRegression()
            reg.fit(row_data.index.values.reshape(-1,1), row_data.values.reshape(-1,1))
            local_window.iloc[row_idx, col_idx] = reg.predict([[local_window.columns[col_idx]]])[0][0]
            continue
        # 行数据不足时用同一列数据拟合
        col_data = local_window.iloc[:, col_idx].dropna()
        if len(col_data) >= 2:
            reg = LinearRegression()
            reg.fit(col_data.index.values.reshape(-1,1), col_data.values.reshape(-1,1))
            local_window.iloc[row_idx, col_idx] = reg.predict([[local_window.index[row_idx]]])[0][0]
    return local_window

3. 标准化后再做全局回归（可选） #

如果坚持用2D线性回归，先对特征做标准化，消除尺度差异的影响：

from sklearn.preprocessing import StandardScaler

def fill_nan_scaled_regression(local_window):
    if not local_window.isnull().values.any():
        return local_window
    
    X_local = local_window.columns.values.copy()
    y_local = local_window.index.values.copy()
    X_local, y_local = np.meshgrid(X_local, y_local)
    
    X_flat = X_local.flatten()
    y_flat = y_local.flatten()
    values_flat = local_window.values.flatten()
    
    non_nan_mask = ~np.isnan(values_flat)
    X_filtered = X_flat[non_nan_mask].reshape(-1,1)
    y_filtered = y_flat[non_nan_mask].reshape(-1,1)
    values_filtered = values_flat[non_nan_mask]
    
    # 标准化特征
    scaler = StandardScaler()
    X_y_scaled = scaler.fit_transform(np.column_stack((X_filtered, y_filtered)))
    
    regressor = LinearRegression()
    regressor.fit(X_y_scaled, values_filtered)
    
    nan_indices = np.argwhere(np.isnan(local_window.values))
    X_nan = local_window.columns.values[nan_indices[:,1]].reshape(-1,1)
    y_nan = local_window.index.values[nan_indices[:,0]].reshape(-1,1)
    X_nan_scaled = scaler.transform(np.column_stack((X_nan, y_nan)))
    
    predicted_values = regressor.predict(X_nan_scaled)
    local_window.iloc[nan_indices[:,0], nan_indices[:,1]] = predicted_values
    
    return local_window

内容的提问来源于stack exchange，提问作者yungpadewon