基于Sci-kit Learn的层级多分类堆叠模型搭建技术咨询

Got it, let's break down how to tackle this hierarchical multi-class classification task with stacked models using your 30k-feature document term matrix (DTM). I’ve worked through similar high-dimensional, hierarchical scenarios before, so here’s a practical, step-by-step guide tailored to your needs:

First off, you can’t train three separate independent models—since Level 2 is tied to Level 1 and Level 3 to Level 2, your pipeline needs to enforce these dependencies. Otherwise, you might end up with predictions like a Level 3 category that doesn’t belong to its predicted Level 2 parent, which breaks your task’s logic.

30k features is a lot for text data, and most are probably redundant noise. Do this preprocessing before building any models:

• Feature Selection: Use SelectKBest with chi-squared test (great for count-based features like DTM) or L1-regularized logistic regression (LogisticRegression(penalty='l1', solver='saga')) to prune irrelevant features. Aim to cut down to 5k-10k meaningful features—this will speed up training and reduce overfitting.
• Normalization/Standardization: If you’re using distance-based models (like SVM or k-NN) in your stack, apply StandardScaler to your features. Tree-based models (XGBoost, Random Forest) don’t need this, but standardizing across the board keeps things consistent for stacking.

We’ll use a hierarchical stacked approach—each layer builds on the previous one’s predictions, while respecting the category dependencies.

3.1 Level 1: Top-Hierarchy Classification #

This is your broadest category layer, so you can treat it as a standard multi-class task first:

• Base Models to Use:
  • LogisticRegression: Fast, handles high-dimensional data well, and L2 regularization (penalty='l2') will keep it from overfitting.
  • MultinomialNB: Perfect for count-based DTM features—super fast and works surprisingly well for text.
  • RandomForestClassifier: Catches non-linear relationships and gives you feature importance insights.
• Stacking Step: Run 5- or 10-fold cross-validation on each base model, then save the predicted class probabilities for each sample. These probabilities become your "meta-features" for the next layer.

3.2 Level 2: Dependent on Level 1 #

Here’s where the hierarchy matters—don’t train a single global Level 2 model. Instead:

• For each Level 1 category, filter your dataset to only include samples that belong to that Level 1 class.
• Train Level 2 models on these filtered subsets. Your input features will be:
  • The original (pruned) DTM features
  • The Level 1 meta-features (probabilities) from the previous step
• Stacking Tip: Again, use cross-validation within each subset to generate Level 2 meta-features (probabilities for each Level 2 category) to pass to Level 3.

3.3 Level 3: Dependent on Level 2 #

Repeat the Level 2 strategy, but now group samples by their Level 2 categories:

• Filter the dataset for each Level 2 class, then train Level 3 models using:
  • Pruned DTM features
  • Level 1 meta-features
  • Level 2 meta-features
• For small subsets (some Level 2 categories might have few Level 3 samples), stick to simpler models like logistic regression to avoid overfitting. For larger subsets, use XGBoost or LightGBM for better performance.

• Manual Cross-Validation for Meta-Features: While sklearn.ensemble.StackingClassifier is handy, manual cross-validation gives you more control over the hierarchical grouping. For each subset, run K-fold CV, predict on the fold’s validation set, and concatenate these predictions to build your meta-feature matrix.
• Meta-Model Choice: For each layer’s final fusion, use a simple linear model like logistic regression or linear SVM. The meta-features are already probability scores from base models, so a linear model is enough to combine them effectively without overcomplicating things.
• Handle Class Imbalance: With 400 categories, imbalance is guaranteed. For every model you train:
  • Use class_weight='balanced' (supported by most scikit-learn models)
  • For tiny minority classes, try SMOTE (but only apply it to the training fold during CV—never touch the validation data!)
  • Evaluate with weighted F1-score instead of accuracy—accuracy is useless for imbalanced hierarchical data.

• Stratified Cross-Validation: Use StratifiedKFold stratified by the Level 3 categories. This ensures your training and test sets have the same hierarchical distribution, so your evaluation is reliable.
• Layer-by-Layer Evaluation: Don’t just look at Level 3’s final score. Check Level 1 and Level 2 accuracy too—if Level 1 is performing poorly, all downstream predictions will be wrong, so you’ll need to tweak that layer first.

• Ignoring Hierarchy: Training three independent models will lead to invalid predictions (e.g., a Level 3 category in the wrong Level 1 branch). Always group samples by their parent category for lower-level training.
• Overcomplicating Small Subsets: If a Level 1 category only has 50 samples for Level 2, don’t use a deep neural network or complex ensemble—stick to logistic regression or Naive Bayes to avoid overfitting.
• Skipping Feature Pruning: 30k features will slow down training and make models prone to overfitting. Spend time on feature selection first—it’s worth the effort.

内容的提问来源于stack exchange，提问作者datadude93