4类方言文本分类模型泛化性能骤降：是否存在过拟合问题？

Let’s start by addressing your core question first: this is not primarily overfitting—it’s almost certainly a case of domain shift, training data quality gaps, and dataset quirks that are dragging down your new Twitter validation accuracy. Here’s a breakdown of what’s happening, plus actionable fixes:

1. Why It’s Not Overfitting #

Overfitting shows up when your model performs drastically better on training data than on distribution-matched test data. Your training set hits 97% accuracy, and your original test set sits at ~90%—that’s a reasonable gap for text classification, not the massive drop you’d expect from overfitting. The real problem is that your Twitter validation data comes from a completely different domain than your training/test sets.

2. The Real Culprits #

Domain Shift #

Your original training data includes manually labeled text for Egypt, Gulf, and Maghribi dialects, plus automatically filtered Hijazi data. But Twitter tweets are a unique domain: they’re shorter, more informal, full of slang, emojis, abbreviations, and possibly code-switching (mixing Arabic with other languages). Your model was trained on cleaner, more structured text, so it can’t generalize to this messier, real-world data.

Training Data Quality Issues (Hijazi) #

Your Hijazi training data was collected via Twitter API with filters, but never manually labeled. That means it’s almost certainly full of noise—users in the target region might speak other dialects, or use Hijazi-specific keywords without actually speaking the dialect. This shows up clearly in your new data’s classification report:

• Hijazi has a high recall (0.83) but terrible precision (0.42). That means your model is labeling almost every possible candidate as Hijazi, even when it’s wrong—because it learned noisy, unreliable features from the unlabeled training data.

Missing Class in Validation Data #

Look at your new dataset’s classification report: Maghribi has a support of 0. That means there are no Maghribi tweets in your 513-sample validation set. Your model can’t predict a class it never sees in the data, and this zero precision/recall is dragging down your overall accuracy significantly.

3. Tweaking Naive Bayes Hyperparameters (Worth Doing, But Not the Fix) #

You’re using the default MultinomialNB settings, which are decent but not optimal. The main hyperparameter to adjust is alpha (Laplace smoothing), which controls how the model handles rare words. A higher alpha reduces overfitting to rare features, while a lower alpha lets the model rely more on specific terms.

Here’s how to tune it with grid search:

from sklearn.model_selection import GridSearchCV

# Define parameter grid for alpha
params = {'clf__alpha': [0.01, 0.1, 0.5, 1.0, 2.0, 5.0]}
grid_search = GridSearchCV(text_clf, params, cv=5, scoring='accuracy')
grid_search.fit(X_train, y_train)

# Use the best estimator
text_clf_optimized = grid_search.best_estimator_
print(f"Best alpha found: {grid_search.best_params_['clf__alpha']}")

You can also improve your CountVectorizer:

• Add Arabic stop words (if you have a reliable list) with stop_words=your_arabic_stopwords
• Try ngram_range=(1,2) to capture two-word phrases, which can help with dialect-specific collocations

4. Actionable Fixes to Boost Twitter Performance #

Fix the Hijazi Training Data #

• Manually label a subset of your Hijazi training data (even 10-20% will help) to clean up noise.
• Use semi-supervised learning: train a base model on the labeled dialects, then use it to predict labels for the unlabeled Hijazi data, and keep only the high-confidence predictions.

Adapt to the Twitter Domain #

• Collect a small set of manually labeled Twitter tweets (even 50-100 per dialect) and fine-tune your model on this data.
• Preprocess your training data to mimic Twitter text: add abbreviations, slang, and emojis (if relevant) to help the model learn domain-specific patterns.

Fix Validation Data Issues #

• If Maghribi tweets don’t exist in your target Twitter dataset, exclude this class from your accuracy calculation (it’s unfair to penalize the model for a class that isn’t present).
• Analyze misclassified tweets: look at which dialects are being mixed up (e.g., Gulf has a recall of only 0.37) and identify unique terms or patterns that your model is missing. Add these to your training data or feature set.

内容的提问来源于stack exchange，提问作者John Sall