基于TensorFlow的一类自编码器异常检测问题：全测试样本被判定为正常

Hey there, let's dig into why your one-class autoencoder is flagging all test samples as normal—this is a super common pitfall with these models, so let's break it down step by step!

1. Your threshold selection logic is too strict #

You're using the maximum reconstruction error from the training set as your threshold. That means a test sample only gets flagged as abnormal if its error is higher than every single normal training sample. If your training data has even a small amount of natural variance in reconstruction error, or your abnormal samples don't have drastically higher errors, this threshold will miss all anomalies.

• Fix: Swap the max value for a percentile of the training error distribution (like 95th or 99th percentile). This accounts for natural variation in normal data while still catching true outliers. Example code:

  import numpy as np
  # Assume train_recon_errors is your array of reconstruction errors from training data
  threshold = np.percentile(train_recon_errors, 99)  # Use 99th percentile as threshold
  

2. The autoencoder isn't learning normal data well enough #

If your model hasn't fully captured the patterns in normal samples, its reconstruction errors for all data (normal and abnormal) will be high and indistinct. This makes the threshold useless for separating classes.

• Fixes:
  • Tweak the model structure: Add more layers/neurons, or switch to a architecture suited for your data type (e.g., convolutional autoencoders for images).
  • Adjust training parameters: Increase training epochs, tune the learning rate, or try a different loss function (MAE can be more robust to small variations than MSE).
  • Double-check preprocessing: Ensure training and test data use identical scaling/normalization (always fit scalers on training data only!).

3. Abnormal samples are too similar to normal ones #

If your abnormal samples overlap heavily with the normal data distribution, the autoencoder will still be able to reconstruct them well, keeping their errors below your threshold.

• Fixes:
  • Verify your labels: Make sure the samples marked as "abnormal" actually represent the kind of outlier you're trying to detect.
  • Try feature engineering: Extract more discriminative features to feed into the autoencoder, or test other anomaly detection methods (like Isolation Forest or One-Class SVM) to rule out data-specific issues.

4. There's a bug in your evaluation logic #

It's easy to mix up threshold logic or misalign labels. Double-check that you're:

• Flagging samples as abnormal (1) when their error exceeds the threshold, not the other way around
• Using the same label convention (abnormal=1, normal=0) for both ground truth and predictions
• Calculating reconstruction error correctly (e.g., using the same loss function you trained with)

Quick code sanity check:

# Assume test_recon_errors is your test set reconstruction errors, y_true is ground truth labels
predictions = [1 if err > threshold else 0 for err in test_recon_errors]

from sklearn.metrics import precision_recall_fscore_support
precision, recall, fscore, _ = precision_recall_fscore_support(y_true, predictions, average='binary')

内容的提问来源于stack exchange，提问作者user3104352