如何使用MDLP_Discretizer处理CSV数据集并解决IndexError报错

我有一个需要处理CSV文件的程序，要把文件转换成可用数据集。我参考了热门Python教程里的鸢尾花数据集实践示例，想把示例里的datasets.load_iris()替换成本地CSV文件'A1-md.csv'的读取方法。

预期效果 #

程序能正常处理CSV并成功加载数据。

实际效果 #

程序运行失败，报错信息如下：

Traceback (most recent call last):
  File ".\e_disc.py", line 43, in <module>
    main()
  File ".\e_disc.py", line 27, in main
    discretizer.fit(X_train, y_train)
  File "D:\Data Mining\assignment_1\entropy_disc_test\MDLP.py", line 58, in fit
    if len(self._col_idx) != self._data_raw.shape[1]:  # some columns will not be discretized
IndexError: tuple index out of range

可正常运行的参考代码 #

import numpy as np
from sklearn import datasets
from sklearn.model_selection import train_test_split
from MDLP import MDLP_Discretizer

def main():

    ######### USE-CASE EXAMPLE #############

    #read dataset
    dataset = datasets.load_iris()
    print(dataset)
    X, y = dataset['data'], dataset['target']
    feature_names, class_names = dataset['feature_names'], dataset['target_names']
    numeric_features = np.arange(X.shape[1])  # all fetures in this dataset are numeric. These will be discretized
    print(f'numeric_feature type {type(numeric_features)}')
    # Split between training and test
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)

    # #Initialize discretizer object and fit to training data
    discretizer = MDLP_Discretizer(features=numeric_features)
    discretizer.fit(X_train, y_train)
    X_train_discretized = discretizer.transform(X_train)

    #apply same discretization to test set
    X_test_discretized = discretizer.transform(X_test)

    #Print a slice of original and discretized data
    print('Original dataset:\n%s' % str(X_train[0:5]))
    print('Discretized dataset:\n%s' % str(X_train_discretized[0:5]))

    #see how feature 0 was discretized
    print('Feature: %s' % feature_names[0])
    print('Interval cut-points: %s' % str(discretizer._cuts[0]))
    print('Bin descriptions: %s' % str(discretizer._bin_descriptions[0]))

if __name__ == '__main__':
    main()

我的实现代码 #

import numpy as np
import pandas as pd
from sklearn import datasets
from sklearn.model_selection import train_test_split
from MDLP import MDLP_Discretizer


def main():

    ######### USE-CASE EXAMPLE #############

    #read dataset
    dataset = pd.read_csv('A1-dm.csv')
    print(dataset)
    X, y = dataset['A1'].to_numpy(), dataset['Class'].to_numpy()
    print(dataset)
    feature_names, class_names = dataset['A1'].to_numpy(), dataset['Class'].to_numpy()
    numeric_features = X  # all fetures in this dataset are numeric. These will be discretized
    print(numeric_features)
    print(f'numeric_feature type {type(numeric_features)}')
    # #Split between training and test
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)

    print(numeric_features)
    # #Initialize discretizer object and fit to training data
    discretizer = MDLP_Discretizer(features=numeric_features)
    discretizer.fit(X_train, y_train)
    X_train_discretized = discretizer.transform(X_train)

    # #apply same discretization to test set
    X_test_discretized = discretizer.transform(X_test)

    # #Print a slice of original and discretized data
    print('Original dataset:\n%s' % str(X_train[0:5]))
    print('Discretized dataset:\n%s' % str(X_train_discretized[0:5]))

    # #see how feature 0 was discretized
    print('Feature: %s' % feature_names[0])
    print('Interval cut-points: %s' % str(discretizer._cuts[0]))
    print('Bin descriptions: %s' % str(discretizer._bin_descriptions[0]))

if __name__ == '__main__':
    main()

待处理的数据集格式 #

A1,A2,A3,Class
2,0.4631338,1.5,3
8,0.7460648,3.0,3
6,0.264391038,2.5,2
5,0.4406713,2.3,1
2,0.410438159,1.5,3
2,0.302901816,1.5,2
6,0.275869396,2.5,3
8,0.084782428,3.0,3
2,0.53226533,1.5,2

错误根因 #

你触发的IndexError: tuple index out of range报错核心原因是输入的特征矩阵格式不符合要求：

• 鸢尾花示例中datasets.load_iris()返回的特征矩阵X是二维数组，维度为(样本数量, 特征数量)，所以可以通过shape[1]获取特征数
• 你的代码里只取了A1一列，转成numpy数组后是一维数组，维度仅为(样本数量,)，没有第二个维度，调用shape[1]时自然就会报错

修改要点 #

• 特征读取要覆盖所有需要离散化的列，保持二维结构
• numeric_features参数需要传入的是要离散化的特征索引，不是特征数值
• 特征名、类别名直接取表头和类别去重值即可，不要和特征/标签值混淆

修改后完整可运行代码 #

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from MDLP import MDLP_Discretizer

def main():
    # 读取数据集
    dataset = pd.read_csv('A1-dm.csv')
    # 提取所有特征列，保持二维数组结构（样本数*特征数）
    X = dataset[['A1', 'A2', 'A3']].values
    y = dataset['Class'].values
    # 对齐鸢尾花示例的属性格式
    feature_names = dataset.columns[:3].tolist()
    class_names = dataset['Class'].unique().tolist()
    # 传入需要离散化的特征索引，和示例逻辑一致
    numeric_features = np.arange(X.shape[1])
    
    # 拆分训练集测试集
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)

    # 初始化离散器并拟合
    discretizer = MDLP_Discretizer(features=numeric_features)
    discretizer.fit(X_train, y_train)
    X_train_discretized = discretizer.transform(X_train)
    X_test_discretized = discretizer.transform(X_test)

    # 输出结果
    print('原始训练集前5行:\n%s' % str(X_train[0:5]))
    print('离散化后训练集前5行:\n%s' % str(X_train_discretized[0:5]))

    print('第一个特征名称: %s' % feature_names[0])
    print('第一个特征切分点: %s' % str(discretizer._cuts[0]))
    print('第一个特征分箱描述: %s' % str(discretizer._bin_descriptions[0]))

if __name__ == '__main__':
    main()

内容的提问来源于stack exchange，提问作者Evan Gertis