训练集与验证集准确率高，但分类报告表现极差的CV模型问题

问题背景 #

开发计算机视觉神经网络，首次训练后出现矛盾结果：

• 训练日志显示训练集准确率约99%、验证集准确率约94%
• 测试集分类报告准确率仅22%，训练集分类报告准确率仅23%，与训练日志结果严重不符

测试集分类报告 #

precision    recall  f1-score   support

           0       0.02      0.01      0.02        71
           1       0.29      0.29      0.29       483
           2       0.19      0.17      0.18       273
           3       1.00      0.00      0.00         6
           4       0.24      0.25      0.25       401
           5       0.05      0.07      0.06        57
           6       0.00      0.00      0.00        14
           7       0.17      0.19      0.18       253
           8       0.00      0.00      0.00         8

    accuracy                           0.22      1566    
   macro avg       0.22      0.11      0.11      1566 
weighted avg       0.22      0.22      0.22      1566

训练集分类报告 #

precision    recall  f1-score   support

           0       0.03      0.03      0.03       504
           1       0.32      0.32      0.32      3377
           2       0.17      0.16      0.17      1911
           3       0.00      0.00      0.00        44
           4       0.26      0.26      0.26      2808
           5       0.05      0.06      0.06       397
           6       0.01      0.01      0.01        98
           7       0.17      0.19      0.18      1768
           8       0.00      0.00      0.00        56

    accuracy                           0.23     10963
   macro avg       0.11      0.11      0.11     10963
weighted avg       0.23      0.23      0.23     10963

模型代码 #

import tensorflow as tf
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense, Flatten, Dropout
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras.preprocessing.image import ImageDataGenerator
from sklearn.metrics import classification_report
import numpy as np
from keras.optimizers import Adam

# Defining paths to folders with train, test, and validation data
train_data = 'Train'
validation_data = 'Validation'
test_data = 'Test'

# Setting image width and height
img_width, img_height = 224, 224

# Creating ImageDataGenerator for training data with data augmentation
train_datagen = ImageDataGenerator()
train_generator = train_datagen.flow_from_directory(
    train_data,
    target_size=(img_width, img_height),
    batch_size=32,
    class_mode='categorical',
    shuffle=True
)
# Creating ImageDataGenerator for validation data
validation_datagen = ImageDataGenerator()
validation_generator = validation_datagen.flow_from_directory(
    validation_data,
    target_size=(img_width, img_height),
    batch_size=32,
    class_mode='categorical',
    shuffle=True
)

test_datagen = ImageDataGenerator()
test_generator = test_datagen.flow_from_directory(
    test_data,
    target_size=(img_width, img_height),
    batch_size=32,
    class_mode='categorical',
    shuffle=False
)

# Creating a Sequential model
model = Sequential()

# Using a pretrained ResNet50 model with weights from 'imagenet'
pretrained_model = tf.keras.applications.resnet.ResNet50(
                    include_top=False,
                    weights='imagenet',
                    input_shape=(img_width, img_height, 3),
                    pooling='avg',
                    classes=9
                    )

#Freezing the layers of the pretrained model so they won't be trained
for layer in pretrained_model.layers:
    layer.trainable = False

# Adding the pretrained ResNet50 model to our sequential model
model.add(pretrained_model)

# Flattening the output of the pretrained model
model.add(Flatten())

# Adding a dense layer with 1024 units and ReLU activation
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.3))

# Adding the output layer with 9 units (one for each class) and softmax activation
model.add(Dense(9, activation='softmax'))

custom_adam = Adam(learning_rate=0.001)
# Compiling the model with the Adam optimizer and categorical_crossentropy loss
model.compile(optimizer=custom_adam, loss='categorical_crossentropy', metrics=['accuracy'])

# Setting up callbacks for early stopping and model checkpointing
early_stopping = EarlyStopping(monitor='val_accuracy', patience=10)
model_checkpoint = ModelCheckpoint('weightsResNet50test.hdf5', monitor='val_loss', save_best_only=True)

# Training the model using the ImageDataGenerators and saving the training history
history = model.fit(train_generator, validation_data=validation_generator, epochs=1000)

# Plotting the training and validation accuracy over epochs
fig1 = plt.gcf()
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.axis(ymin=0.4, ymax=1)
plt.grid()
plt.title('Model Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epochs')
plt.legend(['train', 'validation'])
plt.show()

predictions = model.predict(test_generator)
test_labels = test_generator.classes
class_labels = np.argmax(predictions, axis=1)
report = classification_report(test_labels, class_labels, zero_division=1)
print(report)

排查与修复方案 #

1. 标签映射不匹配（最可能原因） #

flow_from_directory会按文件夹名称的字典序自动生成class_indices映射，如果训练、验证、测试集的类别文件夹顺序不一致，会导致模型预测的类别索引和真实标签索引完全不对应，分类报告结果无效。

验证操作：

print("训练集类别映射:", train_generator.class_indices)
print("测试集类别映射:", test_generator.class_indices)

如果两者输出的类别与索引对应关系不一致，必须统一：

• 手动指定类别顺序，确保三个数据集的classes参数一致：

# 按实际类别名称定义顺序
class_order = ['class0', 'class1', 'class2', 'class3', 'class4', 'class5', 'class6', 'class7', 'class8']

train_generator = train_datagen.flow_from_directory(
    train_data,
    target_size=(img_width, img_height),
    batch_size=32,
    class_mode='categorical',
    shuffle=True,
    classes=class_order
)

validation_generator = validation_datagen.flow_from_directory(
    validation_data,
    target_size=(img_width, img_height),
    batch_size=32,
    class_mode='categorical',
    shuffle=True,
    classes=class_order
)

test_generator = test_datagen.flow_from_directory(
    test_data,
    target_size=(img_width, img_height),
    batch_size=32,
    class_mode='categorical',
    shuffle=False,
    classes=class_order
)

2. 训练回调未生效，导致无效训练 #

你定义了EarlyStopping和ModelCheckpoint但未传入model.fit，模型会强制训练1000个epoch，大概率出现梯度消失或模型退化：

• 冻结ResNet50所有层后，仅训练全连接层，学习率0.001过高，后期权重震荡，模型会倾向于预测占比最高的类别（如分类报告中的类别1），导致训练日志的加权准确率虚高，但真实分类能力极差。

修复操作：
在model.fit中添加回调参数：

history = model.fit(
    train_generator,
    validation_data=validation_generator,
    epochs=1000,
    callbacks=[early_stopping, model_checkpoint]
)

3. 模型训练策略不合理 #

冻结全部预训练层，仅训练全连接层，无法让预训练模型适配你的数据集，导致特征提取能力不足：

• 解冻ResNet50的顶层（如最后20层）进行微调，提升特征适配性
• 降低学习率至0.0001，避免微调时破坏预训练权重

修复操作：

# 解冻ResNet50最后20层
for layer in pretrained_model.layers[-20:]:
    layer.trainable = True

# 重新编译模型，微调时使用更小的学习率
custom_adam = Adam(learning_rate=0.0001)
model.compile(optimizer=custom_adam, loss='categorical_crossentropy', metrics=['accuracy'])

4. 验证数据加载正确性 #

确认训练集是否被正确加载：

• 检查train_generator.samples是否等于10963（训练集分类报告中的总样本数）
• 打印训练集的批量数据，确认样本与标签对应：

x_batch, y_batch = train_generator.next()
print("批量标签的one-hot编码:", y_batch[0])
print("对应类别索引:", np.argmax(y_batch[0]))

内容的提问来源于stack exchange，提问作者Rafał Wyszyński