如何在sklearn的LeaveOneOut+GridSearchCV/cross_validate中用混淆矩阵自定义评分？

针对你遇到的问题——LeaveOneOut单fold仅1个样本无法计算有效指标，需汇总所有fold结果后计算整体指标，以下分两种场景给出具体实现方案：

一、GridSearchCV调参（用整体LOO指标选择最优参数） #

GridSearchCV默认按fold平均评分，但我们需要基于所有LOO样本的预测结果计算单一指标来选参。最可靠的方式是手动遍历所有参数组合，对每个组合单独执行LOO交叉验证，收集全部预测结果后计算整体指标。

示例代码：

import numpy as np
from sklearn.model_selection import LeaveOneOut
from sklearn.svm import SVC  # 替换为你使用的模型

# 定义参数网格和数据集
param_grid = [
    {'C': [0.1, 1, 10], 'kernel': ['linear']},
    {'C': [0.1, 1, 10], 'kernel': ['rbf'], 'gamma': [0.01, 0.1, 1]}
]
X = ...  # 你的特征矩阵
y = ...  # 二分类标签（0/1）

loo = LeaveOneOut()
best_f1 = -1.0
best_params = None
best_metrics = {}

# 遍历每个参数组合
for params in param_grid:
    model = SVC(**params)
    y_pred_list = []
    y_true_list = []
    
    # 执行LeaveOneOut交叉验证
    for train_idx, test_idx in loo.split(X):
        X_train, X_test = X[train_idx], X[test_idx]
        y_train, y_test = y[train_idx], y[test_idx]
        
        model.fit(X_train, y_train)
        y_pred = model.predict(X_test)
        
        y_pred_list.append(y_pred[0])
        y_true_list.append(y_test[0])
    
    # 转换为数组计算混淆矩阵组件
    y_pred = np.array(y_pred_list)
    y_true = np.array(y_true_list)
    
    tp = np.sum((y_true == 1) & (y_pred == 1))
    tn = np.sum((y_true == 0) & (y_pred == 0))
    fp = np.sum((y_true == 0) & (y_pred == 1))
    fn = np.sum((y_true == 1) & (y_pred == 0))
    
    # 计算整体指标
    sensitivity = tp / (tp + fn) if (tp + fn) != 0 else 0.0
    specificity = tn / (tn + fp) if (tn + fp) != 0 else 0.0
    f1 = 2 * tp / (2 * tp + fp + fn) if (2 * tp + fp + fn) != 0 else 0.0
    
    # 更新最优参数
    if f1 > best_f1:
        best_f1 = f1
        best_params = params
        best_metrics = {
            'sensitivity': sensitivity,
            'specificity': specificity,
            'f1': f1
        }

print(f"最优参数组合: {best_params}")
print(f"整体LOO交叉验证指标: {best_metrics}")

二、cross_validate评估单一模型（计算整体LOO指标） #

直接收集所有LOO测试样本的预测结果，再计算整体指标，有两种实现方式：

方法1：用cross_val_predict快速获取所有预测 #

import numpy as np
from sklearn.model_selection import cross_val_predict, LeaveOneOut
from sklearn.svm import SVC

X = ...
y = ...
model = SVC(C=1, kernel='rbf', gamma=0.1)  # 待评估模型

# 获取所有LOO测试样本的预测结果
y_pred = cross_val_predict(model, X, y, cv=LeaveOneOut())

# 计算混淆矩阵组件
tp = np.sum((y == 1) & (y_pred == 1))
tn = np.sum((y == 0) & (y_pred == 0))
fp = np.sum((y == 0) & (y_pred == 1))
fn = np.sum((y == 1) & (y_pred == 0))

# 计算整体指标
sensitivity = tp / (tp + fn) if (tp + fn) != 0 else 0.0
specificity = tn / (tn + fp) if (tn + fp) != 0 else 0.0
f1 = 2 * tp / (2 * tp + fp + fn) if (2 * tp + fp + fn) != 0 else 0.0

print(f"灵敏度: {sensitivity:.4f}")
print(f"特异度: {specificity:.4f}")
print(f"F1分数: {f1:.4f}")

方法2：手动遍历fold（更灵活，适合自定义逻辑） #

import numpy as np
from sklearn.model_selection import LeaveOneOut
from sklearn.svm import SVC

X = ...
y = ...
model = SVC(C=1, kernel='rbf', gamma=0.1)

loo = LeaveOneOut()
tp = tn = fp = fn = 0

for train_idx, test_idx in loo.split(X):
    X_train, X_test = X[train_idx], X[test_idx]
    y_train, y_test = y[train_idx], y[test_idx]
    
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)[0]
    y_true = y_test[0]
    
    if y_true == 1 and y_pred == 1:
        tp += 1
    elif y_true == 0 and y_pred == 0:
        tn += 1
    elif y_true == 0 and y_pred == 1:
        fp += 1
    else:
        fn += 1

# 计算指标
sensitivity = tp / (tp + fn) if (tp + fn) != 0 else 0.0
specificity = tn / (tn + fp) if (tn + fp) != 0 else 0.0
f1 = 2 * tp / (2 * tp + fp + fn) if (2 * tp + fp + fn) != 0 else 0.0

print(f"灵敏度: {sensitivity:.4f}")
print(f"特异度: {specificity:.4f}")
print(f"F1分数: {f1:.4f}")

注意事项 #

不推荐用全局变量/类累积的方式给GridSearchCV传自定义scorer，因为GridSearchCV会为每个参数组合独立运行交叉验证，共享计数器会导致不同参数组合的结果混淆，手动遍历参数组合的方式更直观可控。

内容的提问来源于stack exchange，提问作者Thao Nguyen