GridSearchCV搭配多类SVM运行过久问题排查与优化咨询

我的多类SVM模型包含3个类别，每类24条数据，单独训练仅需1分钟。但加入GridSearchCV做超参数调优后，模型跑了一整天都没结束，也没报错。开启verbose=1后看到输出「Fitting 4 folds for each of 6160 candidates, totalling 24640 fits」，移除GridSearchCV后训练又恢复到1分钟。请问怎么解决这个问题，或者有没有更优的替代方案？

1. 砍掉无效超参数组合 #

你的参数网格里包含大量无意义的组合，做了很多无用功：

• degree仅对poly核有效，却和linear/rbf/sigmoid核一起搜索
• gamma对linear核无效，属于冗余计算
• verbose是训练输出控制参数，不需要放进搜索网格

优化后的参数网格按核函数分组，只搜索对应有效参数：

param_grid = [
    # 线性核
    {'kernel': ['linear'], 'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]},
    # RBF核
    {'kernel': ['rbf'], 'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000],
     'gamma': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]},
    # 多项式核（degree取2-5足够，0/1无实际意义）
    {'kernel': ['poly'], 'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000],
     'gamma': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
     'degree': [2,3,4,5]}
]

参数组合数从6160骤降到427，计算量直接减到原来的1/14。

2. 用RandomizedSearchCV替代全网格搜索 #

如果追求更快的调参速度，RandomizedSearchCV可以在固定迭代次数内随机采样参数组合，比全网格搜索高效得多，同时也能找到接近最优的参数：

from sklearn.model_selection import RandomizedSearchCV
import numpy as np

param_dist = {
    'C': np.logspace(-3, 3, 7),
    'kernel': ['linear', 'rbf', 'poly'],
    'gamma': np.linspace(0.1, 1.0, 10),
    'degree': np.arange(2,6),
    'decision_function_shape': ['ovo','ovr']
}
# 仅搜索100次组合，远少于全网格的6160次
random_search = RandomizedSearchCV(SVC(), param_distributions=param_dist,
                                   n_iter=100, cv=4, verbose=1, random_state=42, n_jobs=-1)
random_search.fit(X_train, y_train)

3. 把特征工程放进Pipeline #

原代码中GridSearchCV在原始X_train上训练，后续又用特征选择后的数据集，存在冗余计算，还可能导致交叉验证的数据泄漏。将标准化、特征选择、SVM整合进Pipeline，让GridSearchCV在统一流程中运行：

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('feature_sele', SelectFromModel(RidgeClassifier())),
    ('svm', SVC(decision_function_shape='ovr'))
])

# 参数名需和Pipeline步骤对应，比如svm__C
param_grid = [
    {'svm__kernel': ['linear'], 'svm__C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]},
    {'svm__kernel': ['rbf'], 'svm__C': [0.001, 0.01, 0.1, 1, 10, 100, 1000],
     'svm__gamma': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]}
]
grid_search = GridSearchCV(pipe, param_grid, cv=4, verbose=1, n_jobs=-1)
grid_search.fit(X_train, y_train)

4. 其他小优化 #

• 固定decision_function_shape='ovr'：多分类场景下ovr计算速度比ovo快，效果相差不大
• 开启并行计算：添加n_jobs=-1让GridSearchCV利用所有CPU核心，大幅缩短时间
• 减少不必要的参数：比如sigmoid核实际效果一般，可直接从搜索列表中移除

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import os
from sklearn import svm
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix, classification_report, roc_curve, auc, accuracy_score
from sklearn.pipeline import Pipeline
from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import RidgeClassifier
from itertools import cycle

import warnings
warnings.filterwarnings('ignore')

# 数据读取
data_list = []
df_list = []
dirname = r"C:\Users\punnut\Downloads\final_dataset"
# AD=0 MCI=1 CN=2

for sub_root, sub_dirs, sub_files in os.walk(dirname):
    for tfile in sub_files:
        if tfile.startswith('AD+MCI+CN_1'):
            data = os.path.join(sub_root, tfile)
            df = pd.read_csv(data)
            df_list.append(df)
            print(df)
            print('miss ', df.isnull().sum())
            print(df['Class'].value_counts())
            print(df['Class'].value_counts() / np.cfloat(len(df)))
            print(round(df.describe(), 2))

# 合并多文件数据（如果存在多个文件）
df = pd.concat(df_list, ignore_index=True) if len(df_list) > 1 else df

X = df.drop(['Class'], axis=1)
y = df['Class'] # AD=0 MCI=1 CN=2

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 42)

# 构建完整Pipeline，整合预处理、特征选择、SVM
pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('feature_sele', SelectFromModel(RidgeClassifier())),
    ('svm', SVC(decision_function_shape='ovr'))
])

# 优化后的参数网格
param_grid = [
    # 线性核
    {'svm__kernel': ['linear'], 'svm__C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]},
    # RBF核
    {'svm__kernel': ['rbf'], 'svm__C': [0.001, 0.01, 0.1, 1, 10, 100, 1000],
     'svm__gamma': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]},
    # 多项式核
    {'svm__kernel': ['poly'], 'svm__C': [0.001, 0.01, 0.1, 1, 10, 100, 1000],
     'svm__gamma': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
     'svm__degree': [2,3,4,5]}
]

# 开启并行计算，n_jobs=-1利用所有CPU核心
grid_search = GridSearchCV(pipe, param_grid, cv=4, error_score="raise", verbose=1, n_jobs=-1)
grid_search.fit(X_train, y_train)
print(grid_search.best_params_, "\n")

# 用最优模型预测
y_pred = grid_search.predict(X_test)
y_pred2 = grid_search.decision_function(X_test)

# 评估指标计算（修正原代码中TP/TN/FP/FN的逻辑错误）
cm = confusion_matrix(y_test, y_pred)
cm_df = pd.DataFrame(cm,index = ['0','1','2'],columns = ['0','1','2'])

AD_TP = cm_df.iloc[0][0]
AD_TN = cm_df.iloc[1][1]+cm_df.iloc[2][2]
AD_FP = cm_df.iloc[0][1]+cm_df.iloc[0][2]
AD_FN = cm_df.iloc[1][0]+cm_df.iloc[2][0]

MCI_TP = cm_df.iloc[1][1]
MCI_TN = cm_df.iloc[0][0]+cm_df.iloc[2][2]
MCI_FP = cm_df.iloc[1][0]+cm_df.iloc[1][2]
MCI_FN = cm_df.iloc[0][1]+cm_df.iloc[2][1]

CN_TP = cm_df.iloc[2][2]
CN_TN = cm_df.iloc[0][0]+cm_df.iloc[1][1]
CN_FP = cm_df.iloc[2][0]+cm_df.iloc[2][1]
CN_FN = cm_df.iloc[0][2]+cm_df.iloc[1][2]

print('\nAD_True Positives(TP) = ', AD_TP)
print('AD_True Negatives(TN) = ', AD_TN)
print('AD_False Positives(FP) = ', AD_FP)
print('AD_False Negatives(FN) = ', AD_FN,'\n')

print('MCI_True Positives(TP) = ', MCI_TP)
print('MCI_True Negatives(TN) = ', MCI_TN)
print('MCI_False Positives(FP) = ', MCI_FP)
print('MCI_False Negatives(FN) = ', MCI_FN,'\n')

print('CN_True Positives(TP) = ', CN_TP)
print('CN_True Negatives(TN) = ', CN_TN)
print('CN_False Positives(FP) = ', CN_FP)
print('CN_False Negatives(FN) = ', CN_FN,'\n')
print(classification_report(y_test, y_pred))
print('Confusion matrix\n', cm_df)

AD_classification_accuracy = (AD_TP + AD_TN) / float(AD_TP + AD_TN + AD_FP + AD_FN)
print('AD Confusion Matrix Classification accuracy : {0:0.4f}'.format(AD_classification_accuracy))
MCI_classification_accuracy = (MCI_TP + MCI_TN) / float(MCI_TP + MCI_TN + MCI_FP + MCI_FN)
print('MCI Confusion Matrix Classification accuracy : {0:0.4f}'.format(MCI_classification_accuracy))
CN_classification_accuracy = (CN_TP + CN_TN) / float(CN_TP + CN_TN + CN_FP + CN_FN)
print('CN Confusion Matrix Classification accuracy : {0:0.4f}'.format(CN_classification_accuracy))
print('Model accuracy score with optimized hyperparameters: {0:0.4f}\n'. format(accuracy_score(y_test, y_pred)))

# ROC曲线绘制
fpr = dict()
tpr = dict()
roc_auc = dict()
y_test_dummies = pd.get_dummies(y_test, drop_first=False).values
for i in range(3):
    fpr[i], tpr[i], _ = roc_curve(y_test_dummies[:, i], y_pred2[:, i], pos_label=1)
    roc_auc[i] = auc(fpr[i], tpr[i])
colors = cycle(['blue', 'red', 'green'])
for i, color in zip(range(3), colors):
    plt.plot(fpr[i], tpr[i], color=color, lw=2,
             label='ROC curve of class {0} (area = {1:0.2f})'
             ''.format(i, roc_auc[i]))
plt.plot([0, 1], [0, 1], 'k--', lw=2)
plt.xlim([-0.05, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Roc Curve AD=0 MCI=1 CN=2')
plt.legend(loc="lower right")
plt.show()

内容的提问来源于stack exchange，提问作者Grey