使用PSO优化SVM超参数时模型准确率过低的技术求助

我用SVM构建了分类模型，尝试用PSO及带被动聚集的PSO调优SVM超参数，但调优后模型准确率远低于未调优版本，且得到的gamma值大于1。以下是实现代码及实验结果：

import sys
import numpy as np
from sklearn import svm
from sklearn.metrics import mean_squared_error, accuracy_score, hamming_loss, f1_score, roc_auc_score
from sklearn.preprocessing import LabelEncoder

def pso(n_particles, iterations, dimensions, inertia):
    # Range of SVR's hyperparameters (Particles' search space)
    # C, Epsilon and Gamma
    max_c = 1e4
    min_c = 1e-3
    max_e = 1e-1
    min_e = 1e-8
    max_g = 1e3
    min_g = 1e-3

    # Initializing particles' positions randomly, inside
    # the search space
    x = np.random.rand(n_particles, 1)*(max_c - min_c) + min_c
    #y = np.random.rand(n_particles, 1)*(max_e - min_e) + min_e
    z = np.random.rand(n_particles, 1)*(max_g - min_g) + min_g
    c = np.concatenate((x,z), axis=1)

    # Initializing particles' parameters
    v = np.zeros((n_particles, dimensions))
    c1 = 1
    c2 = 1
    c3 = 2
    p_best = np.zeros((n_particles, dimensions))
    p_best_val = np.zeros(n_particles) + sys.maxsize  
    g_best = np.zeros(dimensions)
    g_best_val = sys.maxsize
    best_iter = np.zeros(iterations)
    R = np.random.rand(n_particles)

    # Initializing regression variables
    p_best_RGS = np.empty((n_particles), dtype = object);
    g_best_RGS = sys.maxsize

    le = LabelEncoder()
    for i in range(iterations):
        for j in range(n_particles):
            # Starting Regression
            clf = svm.SVC(kernel = 'rbf', C = c[j][0], gamma = c[j][1])
            #rgs1 = svm.SVC(C = c[j][0], gamma = c[j][2])
        
            # Fitting the curve
            clf.fit(x_train, y_train)
            y_predict = clf.predict(x_test)
            #print(y_predict)
            #y_pred= sc.fit_transform(y_predict)
            #print(y_pred)
            # Using Mean Squared Error to verify prediction accuracy
            #print(Y_test)
            mse = roc_auc_score(y_test, y_predict) 
            
            # If mse value for that search point, for that particle,
            # is less than its personal best point,
            # replace personal best
            if(mse < p_best_val[j]):   # mse < p_best_val[j]
                # The value below represents the current least Mean Squared Error
                p_best_val[j] = mse
                p_best_RGS[j] = clf
                # The value below represents the current search coordinates for
                # the particle's current least Mean Squared Error found
                p_best[j] = c[j].copy()
                
            # Using auxiliar variable to get the index of the
            # particle that found the configuration with the 
            # minimum MSE value
            aux = np.argmin(p_best_val)        
        
            if(p_best_val[aux] < g_best_val):
                # Assigning Particle's current best MSE to the Group's best    
                g_best_val = p_best_val[aux]

                # Assigning Particle's current best configuration to the Group's best
                g_best = p_best[aux].copy()

                # Group best regressor:
                # the combination of C, Epsilon and Gamma
                # that computes the best fitting curve
                g_best_RGS = p_best_RGS[aux]

        
                rand1 = np.random.random()
                rand2 = np.random.random()
                rand3 = np.random.random()

            # The variable below influences directly the particle's velocity.
            # It can either make it smaller or bigger. 
            w = inertia

            # The equation below represents Particle's velocity, which is
            # the rate of change in its position
            #v[j] = w*v[j] + c1*(p_best[j] - c[j])*rand1 + c2*(g_best - c[j])*rand2 + c3*(R[j] - c[j])*rand3
            v[j] = w*v[j] + c1*(p_best[j] - c[j])*rand1 + c2*(g_best - c[j])*rand2 
            # Change in the Particle's position 
            c[j] = c[j] + v[j]

            # Below is a series of conditions that stop the particles from
            # leaving the search space
            if(c[j][1] < min_g):
                c[j][1] = min_g
            if(c[j][1] > max_g):
                c[j][1] = max_g
            #if(c[j][1] < min_e):
                #c[j][1] = min_e
            #if(c[j][1] > max_e):
                #c[j][1] = max_e
            if(c[j][0] < min_c):
                c[j][0] = min_c
            if(c[j][0] > max_c):
                c[j][0] = max_c
            
     
        # The variable below represents the least Mean Squared Error
        # of the current iteration
        best_iter[i] = g_best_val
                
        print('Best value iteration # %d = %f\n'%(i, g_best_val))

实验结果 #

准确率 #

超参数 #

问题原因分析 #

1. 优化目标完全反向：roc_auc_score是越大越好的评估指标，但代码中初始化p_best_val为极大值sys.maxsize，并用mse < p_best_val[j]作为更新条件，导致PSO一直在寻找AUC最低的超参数，完全违背调优方向，这是核心问题。
2. 超参数搜索范围不合理：gamma的搜索范围设为1e-3到1e3，跨度太大，而SVM默认gamma为scale（计算方式为1/(n_features*X.var())），通常远小于1，过大的gamma会让模型过度拟合训练数据，导致测试准确率下降。
3. PSO实现逻辑错误：
   • rand1、rand2的生成被放在全局最优更新的分支里，只有更新全局最优时才生成新随机数，大部分迭代用旧值，粒子更新逻辑混乱。
   • 被动聚集相关代码被注释，且R变量初始化后无更新逻辑，等于未启用该机制。
   • 直接用测试集评估超参数，未做交叉验证，容易过拟合测试集，得到的超参数泛化性差。
4. 数据预处理缺失：SVM对特征缩放敏感，代码未做标准化/归一化；初始化了LabelEncoder但未实际转换标签，若y为类别字符串，会导致roc_auc_score计算错误。

解决方案 #

1. 修正优化目标：
   • 将p_best_val初始化为极小值（如-1），更新条件改为if(mse > p_best_val[j])，让PSO寻找AUC最高的超参数。
   • 改用交叉验证结果评估：cross_val_score(clf, x_train, y_train, cv=5, scoring='roc_auc').mean()，避免测试集泄露。
2. 调整超参数搜索范围：
   • gamma范围缩小到1e-5到1，参考默认scale值设置合理区间；C的范围可保留，但建议用对数均匀分布初始化（因为超参数对SVM的影响是对数级的）。
3. 修复PSO逻辑：
   • 将rand1、rand2的生成移到每个粒子的更新循环内，每次更新速度都生成新随机数。
   • 若要启用被动聚集PSO，补全R变量的更新逻辑（比如每隔固定迭代次数重置为随机位置）。
   • 粒子出界后，除了截断位置，还需重置对应维度的速度为0，避免反复出界。
4. 完善数据预处理：
   • 对特征做标准化：

     from sklearn.preprocessing import StandardScaler
     scaler = StandardScaler()
     x_train = scaler.fit_transform(x_train)
     x_test = scaler.transform(x_test)
     

   • 用LabelEncoder转换标签：

     y_train = le.fit_transform(y_train)
     y_test = le.transform(y_test)
     

5. 基准对比：先确认SVM默认参数（svm.SVC()，gamma='scale'）的表现，再对比调优后的结果，确保调优方向正确。

内容的提问来源于stack exchange，提问作者Akshita