Great questions—let's break this down step by step, since you're dealing with a common pain point in large-scale unsupervised anomaly detection for log data.

First off: yes, autoencoders are an excellent choice here, and they’re specifically designed to handle the scale and high dimensionality you’re working with (1.5M rows × 100 cols). Here’s why and how to approach it:

• Autoencoders support batch training: Unlike traditional algorithms like SVM or full LOF, you can feed your data in small batches (e.g., 256-1024 samples per batch) which fits in memory, even for massive datasets. You can also leverage GPU acceleration (via frameworks like TensorFlow/PyTorch) to cut down training time dramatically.
• Choose the right autoencoder variant: For anomaly detection, sparse autoencoders (which penalize hidden layer activations to force the model to learn only critical patterns) or DeepSVDD (Deep One-Class SVM, a neural network-based approach optimized for outlier detection) tend to perform well on log data. Even a simple stacked autoencoder with 2-3 hidden layers can learn meaningful representations of normal log patterns, and anomalies will have higher reconstruction error.

That said, don’t write off traditional algorithms entirely—there are workarounds to make them scale:

• Distributed implementations: Use frameworks like Spark MLlib, which has distributed versions of IsolationForest and OneClassSVM that can handle millions of rows without breaking a sweat. The scikit-learn IsolationForest struggles with large n_estimators on single machines, but Spark’s distributed version scales horizontally.
• Linear SVM alternatives: Instead of kernelized SVMs (which are slow on big data), use LinearSVC or SGDClassifier with a one-class setting in scikit-learn. These support batch training, n_jobs for multi-core processing, and are orders of magnitude faster than non-linear SVMs.
• Dimensionality reduction first: Apply a fast, scalable dimensionality reduction technique like random projection or incremental PCA to reduce your 100-dimensional vectors to 20-30 dimensions. This cuts down computation time for traditional algorithms while retaining most critical log pattern information.
• Approximate neighbor algorithms: For LOF/SOS/SOD, use libraries like Annoy or FAISS to build approximate nearest neighbor indexes. This lets you compute outlier scores without calculating exact distances for every sample, making it feasible for 1.5M rows.

Your current approach—injecting heterogeneous text vectors (like Lord of the Rings content) at a 0.3% contamination rate—is totally valid and widely used in unsupervised anomaly detection testing. Here’s why it works:

• Unsupervised models lack labeled data, so injecting known outliers lets you quantify concrete metrics like precision, recall, and ROC-AUC. This is way more reliable than just looking at arbitrary anomaly scores.
• The low contamination rate (0.3%) aligns with real-world log scenarios, where anomalies are rare. Just make sure the injected vectors are truly distinct from your normal log vectors (you can verify this with a t-SNE plot to check clustering).

For additional model selection strategies, consider these:

• Score distribution analysis: For each model, plot the distribution of anomaly scores for normal vs. injected outlier samples. A good model will have a clear separation between the two distributions. You can use statistical tests like the Kolmogorov-Smirnov (KS) test to quantify this separation.
• Semi-supervised fine-tuning: If you can get even a small number of labeled anomalies (e.g., 100-500 samples), use them to fine-tune a semi-supervised model. For example, take a pre-trained autoencoder and add a classification head to distinguish normal vs. anomalous samples—this often boosts performance significantly.
• Ensemble methods: Combine multiple models (e.g., autoencoder + distributed Isolation Forest) to get a more robust anomaly score. Ensemble approaches reduce the risk of missing outliers that a single model might overlook.
• Domain-specific validation: Instead of just using generic heterogeneous text, inject realistic log anomalies (e.g., error messages that don’t appear in your training data, malformed log entries). This tests whether the model can detect the types of anomalies you actually care about in production.
• Stress testing: Evaluate model performance as you scale up the dataset (e.g., from 500k to 1.5M rows) to ensure it doesn’t degrade. Some models (like autoencoders) maintain performance at scale, while others (like naive clustering) might start to fail.

内容的提问来源于stack exchange，提问作者Shreyance Shaw