单类自编码器ROC曲线倒置：高分类指标却低AUC的疑问

我用50000个良性样本训练单类自编码器，测试集包含10000个良性样本和10000个异常样本。模型F1值0.974、Recall值0.992，分类结果为9600个异常样本、10400个良性样本，表现良好，但计算ROC曲线的AUC仅为0.005。使用sklearn的roc_curve()和auc()函数计算，对比预测二值与真实标签完全吻合，但同一数据集训练OCSVM的AUC为0.997，推测自编码器的ROC曲线倒置，但不知如何验证与修正。

自编码器代码 #

from keras.layers import Input, Dense, Dropout
from keras.models import Model 
from keras import regularizers
from keras.callbacks import EarlyStopping
from sklearn.metrics import roc_curve, auc, f1_score, recall_score
from sklearn.svm import OneClassSVM
from sklearn import metrics
import numpy as np
import pickle  # 补充原代码缺失的导入

class Autoencoder:
  
  def __init__(self, encoding_dim=64, activity_regularizer=10e-6):
    self.encoding_dim = encoding_dim
    self.activity_regularizer = activity_regularizer
    self.autoencoder = None
    self.threshold = None
    
  def fit(self, x_train, x_valid, epochs=50, batch_size=256, earlystop_patience=10):
    # Input Shape
    input_dim = x_train.shape[1]

    # Input Layer
    input_layer = Input(shape = (input_dim,))

    # Encoder Layers
    hidden_layer1 = Dense(512, activation='tanh', activity_regularizer=regularizers.l1(self.activity_regularizer))(input_layer)
    hidden_layer1 = Dropout(0.5)(hidden_layer1)

    hidden_layer2 = Dense(256, activation='tanh', activity_regularizer=regularizers.l1(self.activity_regularizer))(hidden_layer1)
    hidden_layer2 = Dropout(0.5)(hidden_layer2)

    encoded = Dense(self.encoding_dim, activation='tanh', activity_regularizer=regularizers.l1(self.activity_regularizer))(hidden_layer2)

    # Decoder Layers
    hidden_layer4 = Dense(256, activation='tanh', activity_regularizer=regularizers.l1(self.activity_regularizer))(encoded)
    hidden_layer4 = Dropout(0.5)(hidden_layer4)

    hidden_layer5 = Dense(512, activation='tanh', activity_regularizer=regularizers.l1(self.activity_regularizer))(hidden_layer4)
    hidden_layer5 = Dropout(0.5)(hidden_layer5)

    decoded = Dense(input_dim, activation='sigmoid')(hidden_layer5)

    # Define autoencoder
    self.autoencoder = Model(inputs = input_layer, outputs = decoded)

    # Compile Autoencoder
    self.autoencoder.compile(optimizer = 'adam', loss = 'mean_squared_error')

    # Early stopping
    earlystop_callback = EarlyStopping(monitor='val_loss', patience=earlystop_patience, verbose=1, mode='min')

    # Train
    self.autoencoder.fit(x_train, x_train, epochs = epochs, batch_size = batch_size, validation_data = (x_valid, x_valid), callbacks=[earlystop_callback])

  def evaluate(self, x_test, true):
    pred = self.autoencoder.predict(x_test)

    # Reconstruction Error
    mse = np.mean(np.power(x_test - pred, 2), axis = 1)
    np.savetxt('mse.csv', mse, delimiter=',')
    # Threshold Calculation
    self.threshold = np.mean(mse)
    print("Threshold: ")
    print(self.threshold)
    print("\n")

    # True Label Calculations for AE
    ae_test = np.where(mse <= self.threshold, 1, -1)
    anomoly_counter = 0
    normal_counter = 0
    np.savetxt('ae_test.csv', ae_test, delimiter=',')

    for val in ae_test:
      if val == -1:
        anomoly_counter += 1
      elif val == 1:
        normal_counter += 1

    # AUC Calculations
    fpr, tpr, _ = roc_curve(true, mse, pos_label=1)
    auc_num = auc(fpr, tpr)


    # F1 Score
    f1 = f1_score(true, ae_test)


    # Recall Score
    recall = recall_score(true, ae_test)

    print('AUC: {:.3f}'.format(auc_num))       
    print('Recall: {:.3f}'.format(recall))
    print('F1 Score: {:.3f}'.format(f1))
    print('Anomaly: {:.3f}'.format(anomoly_counter))
    print('Positive Class: {:.3f}'.format(normal_counter))

    return fpr, tpr

  def save_model(self, model_file):
    with open(model_file, 'wb') as file:
      pickle.dump(self, file)

问题根源 #

ROC曲线倒置的核心原因是良性样本的重构误差（MSE）远低于异常样本，但你在计算ROC时指定pos_label=1（良性为正类），而roc_curve()默认逻辑是：得分越高，越倾向于被判定为正类。但实际情况是，良性样本的MSE越低，异常样本MSE越高，直接将MSE作为得分传入，相当于得分越高越被判定为负类（异常），导致ROC曲线完全反转，AUC趋近于0。

验证方法 #

1. 统计两类样本的MSE分布，确认差异：

   # 假设true中1代表良性，-1代表异常
   benign_mse = mse[true == 1]
   anomaly_mse = mse[true == -1]
   print(f"良性样本平均MSE: {np.mean(benign_mse):.4f}")
   print(f"异常样本平均MSE: {np.mean(anomaly_mse):.4f}")
   

   如果异常样本平均MSE显著高于良性，说明得分逻辑与正类定义不匹配。

2. 查看ROC输出的fpr和tpr数组：如果fpr随阈值升高快速上升，tpr几乎没有变化，即可确认曲线倒置。

修正方案 #

有两种直接的解决方式：

方式一：反转得分 #

将MSE取负，让得分越高越对应正类（良性样本的负MSE值更高）：

# 替换原AUC计算代码
fpr, tpr, _ = roc_curve(true, -mse, pos_label=1)
auc_num = auc(fpr, tpr)

方式二：调整正类标签 #

将异常样本设为正类（pos_label=-1），此时MSE越高越对应正类，符合逻辑：

# 替换原AUC计算代码
fpr, tpr, _ = roc_curve(true, mse, pos_label=-1)
auc_num = auc(fpr, tpr)

额外优化：阈值计算 #

当前用测试集整体MSE均值作为阈值不合理，建议改用训练集良性样本的MSE分布确定阈值（比如取95分位数），避免测试集异常样本干扰：

# 修改fit方法，添加训练集MSE计算和阈值设定
def fit(self, x_train, x_valid, epochs=50, batch_size=256, earlystop_patience=10):
    # ... 原有训练代码 ...
    # 计算训练集的重构误差
    train_pred = self.autoencoder.predict(x_train)
    train_mse = np.mean(np.power(x_train - train_pred, 2), axis=1)
    # 取训练集MSE的95分位数作为阈值
    self.threshold = np.percentile(train_mse, 95)

内容的提问来源于stack exchange，提问作者Edin Aleckovic