图像分类：五千类别不平衡数据的预训练模型微调问题咨询

Hey there! Let's work through your imbalanced image classification challenge when fine-tuning pre-trained models like Inception. I’ve handled similar high-class-count, skewed datasets before, so here’s a practical breakdown of approaches—including addressing the oversampling concerns you mentioned:

Blindly repeating minority samples will lead to overfitting, so you need smart, augmentation-driven oversampling:

• Augmentation Combinations: Apply diverse, class-specific augmentations to minority samples. For example, use random crops, horizontal/vertical flips, brightness/contrast jitter, Gaussian noise, or advanced techniques like CutMix/MixUp. For extremely small classes (≤50 samples), you could even experiment with style-based generative models to synthesize realistic new images (though this is more resource-intensive).
• Weighted Oversampling: Assign sampling probabilities inversely proportional to class size. This avoids over-saturating the model with the absolute smallest classes while still boosting under-represented ones.
• Dynamic Augmentation in Data Loaders: Implement this directly in your data pipeline (e.g., in PyTorch Dataset or TensorFlow tf.data). For minority class samples, apply 2-3 augmentation transforms per sample, while using minimal augmentations for majority classes.

This is often more efficient than sampling alone, as it adjusts the loss penalty to prioritize minority classes:

• Calculate class weights using formulas like total_samples / (num_classes * class_samples) or use scikit-learn's compute_class_weight for balanced weights.
• Example implementation in TensorFlow/Keras:

import numpy as np
from sklearn.utils.class_weight import compute_class_weight

# Compute weights from training labels
class_weights = compute_class_weight(
    class_weight='balanced',
    classes=np.unique(y_train),
    y=y_train
)
class_weight_dict = dict(zip(np.unique(y_train), class_weights))

# Compile model with weighted loss
model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy'],
    class_weight=class_weight_dict
)

• Note: If using CutMix/MixUp, ensure your loss calculation accounts for the weighted blended labels to maintain fairness across classes.

Pre-trained backbones need careful unfreezing to avoid overwhelming minority classes:

• Phase 1: Freeze Backbone, Train Head: Start by freezing the entire Inception backbone and only training your custom classification head. Use class-weighted loss here—this lets the model learn to map pre-trained features to your classes without letting majority classes dominate early training.
• Phase 2: Unfreeze Layers Gradually: Once the classification head stabilizes, unfreeze the top 1-2 blocks of the backbone (the most task-specific layers) and continue training with a reduced learning rate (e.g., 1e-5 vs. the initial 1e-3). Keep using class-weighted loss and add regularization to prevent overfitting.

Imbalanced data makes models prone to overfitting majority classes or augmented minority samples:

• Add Dropout Layers to your classification head (0.3-0.5 dropout rate) and apply L2 regularization to backbone layers during fine-tuning.
• Use Early Stopping monitored on a balanced metric (like macro-F1 score, not accuracy) to halt training when the model stops improving on minority classes.
• Implement Label Smoothing: Convert hard one-hot labels to soft labels (e.g., y = 0.9 * y + 0.1 / num_classes) to reduce the model's overconfidence in majority class predictions.

Accuracy is meaningless for imbalanced datasets—focus on metrics that highlight minority class performance:

• Macro-F1 Score: Averages precision/recall across all classes equally, ensuring minority classes aren't overlooked.
• Weighted-F1 Score: Balances macro-F1 with class size, useful if you care about overall performance but still want to account for imbalance.
• Confusion Matrix: Visualize which minority classes are being misclassified, then adjust augmentations or weights for those specific classes.
• One-vs-Rest AUC-ROC: For multi-class scenarios, this measures how well the model distinguishes each class from all others.

If oversampling feels risky, mix it with other methods or try these:

• Cluster-Based Undersampling: Instead of randomly dropping majority class samples, cluster them and keep only representative samples from each cluster to preserve information.
• Hybrid Sampling: Combine oversampling of minority classes with undersampling of majority classes (e.g., using image-specific variants of SMOTE alongside cluster undersampling).

内容的提问来源于stack exchange，提问作者Tong Shen