多类别文本分类中不平衡数据引发误分类问题咨询

Hey there, let's tackle this imbalanced text classification problem you're facing—especially with that large (and growing) dataset of 8.5M records across 39 classes. First, let's recap where you're at to ground our solutions:

Current Setup Recap #

• Task: Text classification (both class count and record volume will expand)
• Core Issue: Severe class imbalance driving misclassification
• Completed Work:
  • Combined product_title and key_value_pairs columns
  • Cleaned text: removed stopwords/special characters, applied stemming
  • Implemented processing pipelines

Now, here's a structured, scalable approach to fix the misclassification issue and prepare for future growth:

1. Target Class Imbalance (The Root Cause) #

Data-Level Fixes (Avoid Overfitting!) #

• Synthetic Minority Oversampling: Skip naive duplication of minority samples—it leads to overfitting. Instead, use tools like nlpaug to generate realistic synthetic text for underrepresented classes. You can swap synonyms, tweak word order, or use contextual embeddings (like BERT) to create unique, meaningful samples. Focus only on the most imbalanced classes (e.g., those with <1% of total data) to keep this efficient.
• Selective Majority Undersampling: Don't randomly cut down majority classes—use cluster-based undersampling (via imbalanced-learn) to retain only the most representative samples of large classes. Always do this after splitting train/test data to avoid information leakage.
• Class Weighting: This is the easiest win in your pipeline. For scikit-learn models, add class_weight='balanced' to automatically assign weights inversely proportional to class size. For deep learning workflows later, use weighted cross-entropy loss.

2. Optimize Preprocessing for Large, Growing Data #

Your preprocessing needs to stay efficient as your dataset scales:

• Switch to HashingVectorizer: TF-IDF stores a massive vocabulary, which gets unwieldy with 8.5M records. HashingVectorizer hashes tokens into a fixed-size space, cutting memory usage drastically while preserving most signal.
• Parallelize Cleaning: Use spaCy with n_process=-1 or nltk with multiprocessing to speed up stemming/cleaning. Also, cache your preprocessed text (with joblib) so you don't re-run cleaning every time you test a model.
• Lightweight Embeddings (If You Need Semantics): If TF-IDF isn't enough, use pre-trained lightweight embeddings like distilbert-base-uncased—it's 70% smaller than full BERT but retains most performance. Batch-process embeddings to avoid memory overload.

3. Choose Scalable, Imbalance-Aware Models #

Fast, Traditional Models (Great for Initial Iterations) #

• LightGBM/XGBoost: These tree-based models handle imbalance natively with class weights, and are optimized for large datasets. LightGBM is especially fast thanks to histogram-based splitting.
• Linear SVM with Class Weights: Linear SVMs are efficient for high-dimensional text data, and the class_weight='balanced' parameter helps level the playing field for minority classes.

Deep Learning (For Better Semantic Performance) #

If you have compute resources and need higher accuracy:

• Hierarchical Classification: Split your 39+ classes into a hierarchy (e.g., main categories → subcategories). This reduces per-task imbalance and makes adding new classes easier later.
• DistilBERT with Weighted Loss: Use distilBERT for faster training than full BERT, and apply weighted cross-entropy to prioritize minority classes. For future new classes, use few-shot learning to fine-tune only the classification head with small amounts of new data.

4. Use the Right Evaluation Metrics (Accuracy is Misleading!) #

Stop judging performance by accuracy—it favors majority classes. Instead:

• Macro F1-Score: Treats all classes equally, so you'll clearly see how poorly minority classes are performing.
• AUC-PR (Precision-Recall AUC): Better than ROC-AUC for imbalanced data, as it focuses on the minority class's performance.
• Confusion Matrix: Visualize which classes are being mixed up—this can help you spot if minority classes are confused with similar majority classes, letting you add targeted data or features.

内容的提问来源于stack exchange，提问作者outlier