数据集划分训练验证测试集时索引重复问题的轻量解决方法

你的推测完全正确：两次独立调用np.random.choice生成的validation_indices和test_indices大概率存在重复索引，而np.delete遇到重复索引只会执行一次删除操作——这就导致训练集里还保留着那些重复的样本，同时这些样本又出现在验证集和测试集里，最终三个集合的样本总数超过了原数据集的34322条。

下面是两种改动极小的修复方案：

方案一：先一次性抽取验证+测试的所有不重复索引，再拆分 #

只需要替换原函数中生成索引的3行代码，就能彻底避免重复：

X = df.drop("RISK DECISION", axis = 1).values
y = df["RISK DECISION"].values

def train_validation_test_split(X, y, validation_size = 0.1, test_size = 0.1, random_state = None):
    
    if random_state != None:
        np.random.seed(random_state)
    
    n = X.shape[0]
    
    # 改动部分：先选所有验证+测试的不重复索引，再拆分
    total_val_test_count = int(n*(validation_size + test_size))
    val_test_indices = np.random.choice(n, total_val_test_count, replace=False)
    validation_indices = val_test_indices[:int(n*validation_size)]
    test_indices = val_test_indices[int(n*validation_size):]
    
    all_indices = np.concatenate((validation_indices, test_indices))
    
    X_validation = X[validation_indices]
    y_validation = y[validation_indices]
    X_test = X[test_indices]
    y_test = y[test_indices]
    
    X_train = np.delete(X, all_indices, axis = 0)
    y_train = np.delete(y, all_indices, axis = 0)
    
    return(X_train, X_validation, X_test, y_train, y_validation, y_test)

方案二：选完验证索引后，从剩余样本中抽取测试索引 #

这种方式也能保证索引无重复，只需要修改test_indices的生成逻辑：

X = df.drop("RISK DECISION", axis = 1).values
y = df["RISK DECISION"].values

def train_validation_test_split(X, y, validation_size = 0.1, test_size = 0.1, random_state = None):
    
    if random_state != None:
        np.random.seed(random_state)
    
    n = X.shape[0]
    
    validation_indices = np.random.choice(n, int(n*validation_size), replace = False)
    # 改动部分：从非验证样本的索引中选测试集
    remaining_indices = np.setdiff1d(np.arange(n), validation_indices)
    test_indices = np.random.choice(remaining_indices, int(n*test_size), replace = False)
    
    all_indices = np.concatenate((validation_indices, test_indices))
    
    X_validation = X[validation_indices]
    y_validation = y[validation_indices]
    X_test = X[test_indices]
    y_test = y[test_indices]
    
    X_train = np.delete(X, all_indices, axis = 0)
    y_train = np.delete(y, all_indices, axis = 0)
    
    return(X_train, X_validation, X_test, y_train, y_validation, y_test)

两种方案都能保证三个集合的样本完全无重叠，总数刚好等于原数据集的34322条，而且对原函数的改动都控制在3行以内。

内容的提问来源于stack exchange，提问作者Davide