DNN模型训练准确率高但测试准确率低问题排查求助

Hey Richard, let's break down why your DNN is crushing the training set but failing on the test data—even though you ruled out overfitting (which makes total sense given your 700k-sample training set and tiny model), there are several key areas to investigate:

1. Double-Check Label Handling Consistency #

Looking at your parse_function, you’re grabbing the second element of the label array with parsed_example['label'][1]. This is a common spot for silent errors:

• Are you 100% sure both training and test TFRecords store labels in the same order? For example, could the test set’s labels be [target, non-target] while the training set uses the reverse? If so, your model is learning to predict the wrong class entirely—explaining the plummeting test accuracy.
• When loading the test set into X_test and y_test, print a handful of y_test values to confirm they’re valid 0/1 binary labels matching your task.
• You cast the label to tf.int32 with label_t = tf.cast(...) but never use that variable—clean this up to avoid accidental bugs down the line.

2. Investigate Train-Test Data Distribution Shift #

Even a small mismatch in data preprocessing or class distribution can tank test performance:

• Feature Normalization: Did you apply identical normalization to both training and test data? Your parse_function decodes raw float64 data, but if training TFRecords contain scaled/standardized features while the test set stores raw values, the model will be thrown off. Compare the mean/std of a few features from train_dataset and X_test to check alignment.
• Class Ratio Alignment: You noted a 0.37:0.63 positive-to-negative ratio, but is this exactly the same in both train and test sets? If the test set has a drastically different split (e.g., 0.1:0.9), your model’s bias toward the majority class will make test accuracy look terrible. Calculate the class distribution of y_test to confirm.
• Feature Shape: Verify that X_test samples have the exact (28,20,1) shape your model expects. Print X_test[0].shape—if the channel dimension is missing or the size is wrong, the model will make garbage predictions.

3. Debug Your Data Loading and Evaluation Pipeline #

Your training and test pipelines have key differences that might be causing issues:

• Test Data Conversion: Manually converting the test tf.data.Dataset to numpy arrays via a session loop could introduce bugs (e.g., misaligned X_test and y_test). Try evaluating directly using the test dataset instead:

  result = model_dnn.evaluate(test_dataset.batch(32))
  

  This uses the same pipeline structure as training and skips the error-prone conversion step.
• Evaluation Batch Size: Using batch_size=len(X_test) for evaluation can cause memory issues or numerical instability, especially with large test sets. Switch to a smaller batch size (like 32 or 64) to see if accuracy improves.
• Training Data Coverage: Your train_dataset uses repeat(100) and steps_per_epoch=1000—with a batch size of 670, each epoch only processes 670k samples (your full training set is 700k). This means you’re not covering all training data each epoch, though this is unlikely to be the main issue after 10 epochs.

4. Look Beyond Accuracy for Imbalanced Data #

Accuracy is a misleading metric for imbalanced datasets—let’s dig deeper:

• Confusion Matrix: Generate confusion matrices for both train and test sets to see which class the model is misclassifying. For example, if it’s predicting almost all samples as the majority class (63% of your data), training accuracy will look high, but the test set might have more minority-class samples the model fails to detect.
• Adjust Classification Threshold: The default 0.5 threshold for sigmoid outputs isn’t optimal for imbalanced data. Calculate a precision-recall curve for the test set to find a threshold that balances precision and recall, then compute accuracy using that threshold.
• Add Class Weights: Even mild imbalance can benefit from class weights. Modify your fit call to prioritize the minority class:

  class_weight = {0: 1, 1: 0.63/0.37}  # Weight based on your class ratio
  model_dnn.fit(train_dataset, steps_per_epoch=1000, epochs=10, verbose=2, class_weight=class_weight)
  

5. Fix Data Type Mismatches #

Your model uses TensorFlow/Keras’s default float32 for layers, but your parse_function decodes features as float64. While Keras can handle this, mismatched types sometimes cause subtle numerical issues. Update your feature decoding to use float32:

parsed_example['feature'] = tf.cast(tf.decode_raw(parsed_example['feature'], tf.float64), tf.float32)

Start with the label handling and data distribution checks—those are the most common culprits for this kind of training/test performance gap.

内容的提问来源于stack exchange，提问作者Richard Lee