字典转DataFrame时持续报ValueError: 数组长度不一致问题求助

问题描述 #

将字典转换为pandas DataFrame时反复出现ValueError: All arrays must be of the same length报错，已手动确认各数组长度一致，但问题仍存在。

相关代码 #

def metrics_from_pipes(pipes_dict):
     for name, pipeline in pipes_dict.items():
        
        pipeline.fit(X_train, y_train)
        y_pred_val = pipeline.predict(X_val)
        y_pred_train = pipeline.predict(X_train)


train_metrics = {
            'model':list(pipes_dict.keys()),
            'MAE':train_mae,
            'MAPE':train_mape,
            'RMSE':train_rmse,
            'RSquared':train_rsquared
        }
        
        train_metrics_data = pd.DataFrame(train_metrics)
        val_metrics = {
            'model':list(pipes_dict.keys()),
            'MAE':val_mae,
            'MAPE':val_mape,
            'RMSE':val_rmse,
            'RSquared':val_rsquared            
        }
        
        val_metrics_data = pd.DataFrame(val_metrics,)

        #Merging metrics from train and test set
        train_val_metrics = train_metrics_data.merge(val_metrics_data,
                                               on = 'Model',
                                               how = 'left',
                                               suffixes = ('_train', '_val'))
        
        # sorting columns 
        train_val_metrics = train_val_metrics.reindex(columns = ['Model',
                                                               'MAE_train',
                                                                'MAPE_train',
                                                                'RMSE_train',
                                                                'RSquared_train',
                                                                'MAE_val',
                                                                'MAPE_val',
                                                                'RMSE_val',
                                                                'RSquared_val'])
        
    
    return train_val_metrics.set_index('Model').transpose()

# get the metrics table
metrics_table = metrics_from_pipes(pipelines)

报错堆栈信息 #

ValueError                                Traceback (most recent call last)
Cell In[45], line 82
     80     return train_val_metrics.set_index('Model').transpose()
     81 # get the metrics table
---> 82 metrics_table = metrics_from_pipes(pipelines)
     83 #print('Table 1: Base Models Metrics')
     84 #metrics_table.style.background_gradient(cmap = Blues)
     85 metrics_table

Cell In[45], line 50, in metrics_from_pipes(pipes_dict)
     41 # aggregate the performance metric lists into seperate dataframes
     42 train_metrics = {
     43     'model':list(pipes_dict.keys()),
     44     'MAE':train_mae,
   (...)
     47     'RSquared':train_rsquared
     48 }
---> 50 train_metrics_data = pd.DataFrame(train_metrics)
     51 val_metrics = {
     52     'model':list(pipes_dict.keys()),
     53     'MAE':val_mae,
   (...)
     56     'RSquared':val_rsquared            
     57 }
     59 val_metrics_data = pd.DataFrame(val_metrics,)

ValueError: All arrays must be of the same length

检查得到的字典内容 #

({'model': ['Linear Regression',
   'Random Forest Regressor',
   'Gradient Boost Regression',
   'Extra Tree Regressor'],
  'MAE': [829.1023412412194,
   288.33455697065233,
   712.9637267872279,
   0.0010629575741748962],
  'MAPE': [1.0302372135902111,
   0.20937541440883897,
   0.538244903316323,
   6.306697580961048e-07],
  'RMSE': [1120.5542708017374,
   416.48933196590013,
   1012.399201767692,
   0.05804079289490426],
  'RSquared': [0.5598288286601083,
   0.9391916010838417,
   0.6406981997919169,
   0.9999999988190745]},
 {'model': ['Linear Regression',
   'Random Forest Regressor',
   'Gradient Boost Regression',
   'Extra Tree Regressor'],
  'MAE': [855.9254413559535,
   802.5902302175274,
   772.3140648475379,
   839.9018341377154],
  'MAPE': [1.0395487579496652,
   0.5607987708065988,
   0.5438627253681279,
   0.5852285872937784],
  'RMSE': [1148.6549900167981,
   1158.8411708570625,
   1109.6145558003204,
   1223.23337689915],
  'RSquared': [0.5876710102285392,
   0.5803255834810521,
   0.6152231339508221,
   0.5323905190373128]})

问题排查与修复 #

核心问题1：指标变量未正确初始化与填充 #

原代码中train_mae、val_mae等指标变量没有在循环前初始化，且缺失将每个模型的指标值追加到列表的逻辑。这会导致这些变量可能不是长度为4的列表（比如是单个值或未定义），从而与model列表长度不匹配，触发报错。

核心问题2：列名大小写不匹配 #

创建DataFrame时用的是小写'model'，但合并和重排序列时用的是大写'Model'，后续会导致合并失败或列名混乱。

修复后的完整代码 #

import pandas as pd
from sklearn.metrics import mean_absolute_error, mean_absolute_percentage_error, mean_squared_error, r2_score

def metrics_from_pipes(pipes_dict):
    # 初始化所有指标存储列表
    train_mae = []
    train_mape = []
    train_rmse = []
    train_rsquared = []
    val_mae = []
    val_mape = []
    val_rmse = []
    val_rsquared = []
    
    for name, pipeline in pipes_dict.items():
        pipeline.fit(X_train, y_train)
        y_pred_val = pipeline.predict(X_val)
        y_pred_train = pipeline.predict(X_train)
        
        # 计算并收集训练集指标
        train_mae.append(mean_absolute_error(y_train, y_pred_train))
        train_mape.append(mean_absolute_percentage_error(y_train, y_pred_train))
        train_rmse.append(mean_squared_error(y_train, y_pred_train, squared=False))
        train_rsquared.append(r2_score(y_train, y_pred_train))
        
        # 计算并收集验证集指标
        val_mae.append(mean_absolute_error(y_val, y_pred_val))
        val_mape.append(mean_absolute_percentage_error(y_val, y_pred_val))
        val_rmse.append(mean_squared_error(y_val, y_pred_val, squared=False))
        val_rsquared.append(r2_score(y_val, y_pred_val))
    
    # 构建训练集指标字典
    train_metrics = {
        'model': list(pipes_dict.keys()),
        'MAE': train_mae,
        'MAPE': train_mape,
        'RMSE': train_rmse,
        'RSquared': train_rsquared
    }
    train_metrics_data = pd.DataFrame(train_metrics)
    
    # 构建验证集指标字典
    val_metrics = {
        'model': list(pipes_dict.keys()),
        'MAE': val_mae,
        'MAPE': val_mape,
        'RMSE': val_rmse,
        'RSquared': val_rsquared            
    }
    val_metrics_data = pd.DataFrame(val_metrics)
    
    # 合并数据集，使用统一的小写列名
    train_val_metrics = train_metrics_data.merge(val_metrics_data,
                                               on='model',
                                               how='left',
                                               suffixes=('_train', '_val'))
    
    # 重排序列名，最后统一列名为首字母大写
    train_val_metrics = train_val_metrics.reindex(columns=['model',
                                                           'MAE_train',
                                                           'MAPE_train',
                                                           'RMSE_train',
                                                           'RSquared_train',
                                                           'MAE_val',
                                                           'MAPE_val',
                                                           'RMSE_val',
                                                           'RSquared_val'])
    train_val_metrics = train_val_metrics.rename(columns={'model': 'Model'})
    
    return train_val_metrics.set_index('Model').transpose()

# 获取指标表
metrics_table = metrics_from_pipes(pipelines)

修复说明 #

1. 初始化指标列表：在循环前创建空列表，确保每个模型的指标值都能被正确追加，保证所有列表长度与模型数量一致。
2. 补充指标计算逻辑：添加了基于sklearn.metrics的指标计算代码（可根据实际需求替换），确保指标变量被正确填充。
3. 统一列名大小写：合并时使用小写'model'作为键，最后再重命名为'Model'，避免大小写不匹配导致的后续问题。

内容的提问来源于stack exchange，提问作者SirTee12