训练IDS时ColumnTransformer报错ValueError的原因排查

代码片段 #

# Importing the KDCup99 dataset
dataset = pd.read_csv(r'C:\Users\Ahmad\Desktop\Thiese\KDDCup99.csv',on_bad_lines='skip')

X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, 41:42].values

# Spliting the dataset into training and test sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 0)

''' Data Preprocessing '''

# Applying ColumnTransformer to the categorical columns of X_train and X_test
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
ct = ColumnTransformer(transformers = [('encoder', OneHotEncoder(), [1, 2, 3])], remainder = 'passthrough')
X_train = ct.fit_transform(X_train)

错误信息 #

IndexError: index 1 is out of bounds for axis 0 with size 0

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File ~\untitled0.py:29 in <module>
    X_train = ct.fit_transform(X_train)

  File ~\anaconda3\lib\site-packages\sklearn\compose\_column_transformer.py:687 in fit_transform
    self._validate_column_callables(X)

  File ~\anaconda3\lib\site-packages\sklearn\compose\_column_transformer.py:374 in _validate_column_callables
    transformer_to_input_indices[name] = _get_column_indices(X, columns)

  File ~\anaconda3\lib\site-packages\sklearn\utils\__init__.py:384 in _get_column_indices
    raise ValueError(

ValueError: all features must be in [0, -1] or [-0, 0]

报错原因分析 #

• 数据集未成功加载，X_train为空数组（列数为0）：
  错误提示里的axis 0 with size 0和all features must be in [0, -1] or [-0, 0]说明X_train的特征数（列数）为0。大概率是pd.read_csv读取数据时，on_bad_lines='skip'跳过了所有行（包括表头），导致dataset是空的DataFrame，进而X = dataset.iloc[:, :-1].values生成了一个0列的空数组。后续拆分后的X_train自然也是0列，此时尝试访问第1、2、3列必然触发索引越界。

• 不应将DataFrame转为NumPy数组后使用ColumnTransformer：
  ColumnTransformer可以直接处理Pandas DataFrame，且更稳定。你用.values将DataFrame转为NumPy数组后，丢失了列的元信息，同时如果数据加载出现异常（比如空数组），更容易触发这类索引错误。建议直接用DataFrame格式的X进行拆分和预处理，而非转为数组。

内容的提问来源于stack exchange，提问作者Alex