基于APT样本的单类分类器选型及评估指标技术咨询

Hey Luca, great question—working with APT samples and one-class classification is such a niche but critical task, especially given your dataset size (2500+ samples, 300+ features) and multi-category APT setup. Let's break this down for you:

You're already using One-Class SVM, which is a solid choice, but there are several other options that might fit your APT-specific scenario better, depending on your feature distribution and computational constraints:

• Isolation Forest: Perfect for your high-dimensional (300+ features) and large dataset (2500+ samples). It works by randomly isolating outliers (non-target APT samples) instead of modeling the target class directly, making it fast to train and less prone to overfitting on high-dimensional data. You can use scikit-learn's IsolationForest out of the box—just tune the contamination parameter to match the expected ratio of non-target samples.
• Local Outlier Factor (LOF): A density-based method that excels at detecting local anomalies. If a specific APT category has a distinct cluster of feature patterns, LOF will flag samples that deviate from this local density. It's instance-based, so it doesn't require strong distribution assumptions. Note that for very high dimensions, distance calculations can slow things down, but 2500 samples should still be manageable with scikit-learn's LocalOutlierFactor.
• Elliptic Envelope: A robust covariance-based method, ideal if your target APT class has features that roughly follow a Gaussian distribution. It's great for detecting extreme outliers, and scikit-learn's implementation includes a robust version that ignores the most extreme points to avoid skewed covariance estimates. It's lighter on computation than SVM for large datasets.
• Autoencoder-Based Deep One-Class Models: If you have access to moderate computational resources, autoencoders are powerful for learning complex, non-linear feature representations of your target APT samples. Samples that don't belong to the class will have higher reconstruction error. You can build these with TensorFlow/PyTorch, or use a scikit-learn-compatible wrapper like skorch to integrate with your existing workflow. They're especially useful when your 300+ features have complex interdependencies that traditional methods might miss.
• One-Class Naive Bayes: A quick baseline option if your features are discrete or can be approximated as independent Gaussian variables. It's extremely fast to train, even on high-dimensional data, though it may underperform if your APT features have strong correlations. You can adapt scikit-learn's GaussianNB for one-class use by fitting it only on target samples and scoring others based on likelihood.

Standard metrics like accuracy are useless here—one-class classification deals with imbalanced data (your target APT class is a small subset of all samples), so you need metrics that focus on how well you detect non-target samples without misclassifying target ones. Here's what to prioritize:

• AUC-PR (Precision-Recall AUC): This is your go-to metric for imbalanced one-class tasks. Unlike AUC-ROC, which can be misleading when negative samples (non-target APTs) dominate, AUC-PR focuses on the model's ability to correctly identify positive (target) samples while minimizing false positives. It's far more reliable for APT detection where false alarms can waste analyst time.
• Precision@k & Recall@k: If you need to prioritize either minimizing false positives (e.g., only flag the most suspicious samples for manual review) or maximizing detection of non-target samples, use these. For example, Precision@50 tells you how many of the top 50 flagged samples are actually non-target, while Recall@50 tells you what percentage of all non-target samples are in those top 50.
• F1-Score (at Optimal Threshold): Once you tune your model's decision threshold (via cross-validation), the F1-score balances precision and recall. It's a good single-number metric if you need to balance avoiding false positives and missing true anomalies.
• Average Precision (AP): Similar to AUC-PR, but it measures the average precision across all possible recall thresholds. It's great for comparing models that rank samples by anomaly score—higher AP means your model consistently ranks true non-target samples above target ones.
• Custom Threshold-Based Metrics: Depending on your operational needs, you might define metrics like False Positive Rate (FPR) (percentage of target samples misclassified as non-target) and True Positive Rate (TPR) (percentage of non-target samples correctly identified). These are useful if you have strict constraints on how many false alarms you can tolerate.

Also, make sure to use a suitable cross-validation strategy—regular stratified CV won't work for one-class tasks. Instead, use leave-one-out cross-validation (LOOCV) for smaller APT classes, or stratified one-class CV where you split target samples into train/test folds and include a consistent ratio of non-target samples in each fold.

内容的提问来源于stack exchange，提问作者Luca