MobileNetV2食品图像分类迁移学习遇过拟合及验证精度停滞

云原生阿岚

2026-6-25

问题描述

使用MobileNetV2做食品图像分类迁移学习时遭遇严重过拟合：训练准确率达95%以上，训练过程中验证准确率卡在67%左右，但实际预测时验证集准确率仅19%（训练时的val_accuracy与手动预测结果严重不符）。此前用ResNet也存在相同问题，因MobileNet速度更快选择继续使用。相关代码及结果如下：

数据生成代码

# Create an ImageDataGenerator with data augmentation for training
data_generator = keras.preprocessing.image.ImageDataGenerator(
    preprocessing_function=keras.applications.mobilenet.preprocess_input,
    # rescale=1./255,
    validation_split=0.3,
    rotation_range=30,  # Random rotations
    width_shift_range=0.2,  # Horizontal shifts
    height_shift_range=0.2,  # Vertical shifts
    shear_range=0.2,  # Shear transformations
    zoom_range=0.2,  # Zoom
    horizontal_flip=True,  # Horizontal flips
)

# Load and preprocess training data
train_data_flow = data_generator.flow_from_directory(
    dataset_path,
    target_size=(128, 128),  # Resize images to 224x224
    batch_size=32,
    class_mode='categorical',
    subset='training'  # Use training subset
)

# Load and preprocess validation data
val_data_flow = data_generator.flow_from_directory(
    dataset_path,
    target_size=(128, 128),  # Resize images to 224x224
    batch_size=32,
    class_mode='categorical',
    subset='validation'  # Use validation subset
)

模型构建代码

# Load the model from TensorFlow Hub
model_url = "https://www.kaggle.com/models/google/mobilenet-v2/TensorFlow2/035-128-classification/2"
hub_layer = hub.KerasLayer(model_url, input_shape=(128, 128, 3) ,  trainable=True, arguments=dict(batch_norm_momentum=0.997))

# Create a Sequential model with dropout and batch normalization
model = keras.Sequential([
    hub_layer,
    layers.Dropout(0.2),  # Lower dropout rate
    layers.Dense(256, activation='relu'),
    layers.BatchNormalization(),  # Batch normalization
    layers.Dropout(0.2),  # Dropout
    layers.Dense(9, activation='softmax')
])

# Build the Sequential model
model.build((None, 128, 128, 3))

# Summary of the model
model.summary()

训练代码

# Define the EarlyStopping callback
early_stopping_callback = keras.callbacks.EarlyStopping(
    monitor='val_loss',  # Monitor validation loss
    patience=20,          # Number of epochs with no improvement after which training will be stopped
    restore_best_weights=True  # Restore model weights from the epoch with the best value of the monitored quantity
)

# Fit the model with early stopping
history = model.fit(
    train_data_flow,
    validation_data=val_data_flow,
    epochs=100,
    callbacks=[early_stopping_callback],  # Pass the EarlyStopping callback as a list
    shuffle= True
)

训练日志片段

Epoch 1/100
113/113 [==============================] - 73s 641ms/step - loss: 0.5302 - accuracy: 0.8623 - val_loss: 3.2597 - val_accuracy: 0.4723
...
Epoch 8/100
113/113 [==============================] - 71s 625ms/step - loss: 0.3877 - accuracy: 0.9062 - val_loss: 1.4624 - val_accuracy: 0.6630
...
Epoch 44/100
113/113 [==============================] - 71s 632ms/step - loss: 0.2272 - accuracy: 0.9527 - val_loss: 1.3801 - val_accuracy: 0.6760

预测代码与结果

from sklearn.metrics import classification_report

predictions = model.predict(val_data_flow)

# Convert predictions to class labels
predicted_classes = np.argmax(predictions, axis=1)

# Step 4: Evaluate the Model

# Get the true labels from the test data
true_labels = val_data_flow.classes  

# Calculate accuracy
accuracy = np.mean(predicted_classes == true_labels)  

# Accuracy as the score
print("Model Accuracy on Test Set:", accuracy)

# sklearn's classification_report for more detailed metrics
print("Classification Report:")
print(classification_report(true_labels, predicted_classes, target_names=list(val_data_flow.class_indices.keys())))

输出结果：

Model Accuracy on Test Set: 0.19388418998048146
Classification Report:
              precision    recall  f1-score   support

           1       0.32      0.30      0.31       495
           2       0.00      0.00      0.00        21
           3       0.17      0.21      0.19       199
           4       0.13      0.09      0.11       154
           5       0.00      0.00      0.00         8
           6       0.11      0.06      0.08        32
           7       0.11      0.07      0.08       199
           8       0.16      0.19      0.17       231
           9       0.13      0.18      0.15       198

    accuracy                           0.19      1537
   macro avg       0.12      0.12      0.12      1537
weighted avg       0.19      0.19      0.19      1537

解决思路

1. 先修正「训练val_accuracy与实际预测不符」的核心问题

训练时显示67%验证准确率，但手动预测只有19%，本质是验证集数据处理逻辑不一致：

你给验证集也用了带数据增强的ImageDataGenerator，训练时验证集被随机变换，但val_data_flow.classes是原始标签，预测时用增强后的图像与原始标签对应，完全不匹配。

修正方案：

单独创建验证集专用的生成器，仅做预处理，不做任何增强，且关闭shuffle保证顺序一致：

val_data_generator = keras.preprocessing.image.ImageDataGenerator(
    preprocessing_function=keras.applications.mobilenet.preprocess_input
)
val_data_flow = val_data_generator.flow_from_directory(
    dataset_path,
    target_size=(128, 128),
    batch_size=32,
    class_mode='categorical',
    subset='validation',
    shuffle=False  # 必须关闭，保证标签和预测结果一一对应
)

训练和预测都使用这个新的val_data_flow，才能得到真实的验证指标。

2. 针对性解决过拟合问题

（1）调整迁移学习策略：分层解冻预训练层

当前将整个MobileNetV2设为trainable=True，预训练权重被过度修改，极易过拟合。正确步骤：

第一步：冻结所有预训练层，只训练顶部全连接层：

hub_layer = hub.KerasLayer(model_url, input_shape=(128, 128, 3), trainable=False)

训练10-20轮，等验证准确率稳定后，再解冻预训练层的顶部部分（比如最后20层），用极小学习率微调：

hub_layer.trainable = True
# 仅解冻最后20层预训练层
for layer in hub_layer.layers[:-20]:
    layer.trainable = False
# 重新编译，用小学习率
model.compile(optimizer=keras.optimizers.Adam(learning_rate=1e-5),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

（2）强化正则化

提高Dropout比例：将现有Dropout(0.2)改为Dropout(0.5)，可在全连接层前额外加一层Dropout。

给全连接层加L2正则：

layers.Dense(256, activation='relu', kernel_regularizer=keras.regularizers.l2(1e-4)),

缩小全连接层维度：256维容量过大，可改为128或64，减少模型复杂度。

（3）优化数据增强

加大增强力度：提升旋转、平移范围，加入亮度/对比度调整：

data_generator = keras.preprocessing.image.ImageDataGenerator(
    preprocessing_function=keras.applications.mobilenet.preprocess_input,
    validation_split=0.3,
    rotation_range=45,
    width_shift_range=0.3,
    height_shift_range=0.3,
    shear_range=0.2,
    zoom_range=[0.7, 1.3],
    horizontal_flip=True,
    brightness_range=[0.8, 1.2],
    contrast_range=[0.8, 1.2]
)

注意：验证集绝对不能加任何增强，仅做预处理。

（4）处理类别不平衡

从分类报告看，类别2（21样本）、类别5（8样本）严重稀缺，导致模型偏向大类：

对小类别做过采样：复制样本或用数据增强生成更多变体。

训练时使用类别权重：

from sklearn.utils.class_weight import compute_class_weight
class_weights = compute_class_weight('balanced', classes=np.unique(train_data_flow.classes), y=train_data_flow.classes)
class_weights_dict = dict(enumerate(class_weights))
# 训练时传入权重
history = model.fit(
    train_data_flow,
    validation_data=val_data_flow,
    epochs=100,
    callbacks=[early_stopping_callback],
    shuffle=True,
    class_weight=class_weights_dict
)

（5）调整训练细节

降低学习率：初始学习率改用1e-4，微调时用1e-5，避免权重更新过快。
增大Batch Size：硬件允许的话，将batch_size从32改为64/128，提升梯度稳定性。

内容的提问来源于stack exchange，提问作者angryhorse

MobileNetV2食品图像分类迁移学习遇过拟合及验证精度停滞

数据生成代码 #

模型构建代码 #

训练代码 #

训练日志片段 #

预测代码与结果 #

1. 先修正「训练val_accuracy与实际预测不符」的核心问题 #

2. 针对性解决过拟合问题 #

（1）调整迁移学习策略：分层解冻预训练层 #

（2）强化正则化 #

（3）优化数据增强 #

（4）处理类别不平衡 #

（5）调整训练细节 #