基于VGG16迁移学习的药用植物叶分类模型低准确率问题排查

问题背景 #

使用TensorFlow的VGG16进行迁移学习构建药用植物叶片图像识别模型，准确率仅0.05且无法提升，但换用EfficientNetB2可达到0.9875的准确率。相关配置与训练情况如下：

数据集情况 #

• 30个类别，每个类别对应独立文件夹
• 数据集划分：训练集917张、验证集570张、测试集357张
• 图像尺寸调整为256×256，批量大小BATCH_SIZE=16

数据增强配置 #

# Definig ImageDataGenerator With Augmentation
datagen = ImageDataGenerator(
        rotation_range=20,
        width_shift_range=0.1,
        height_shift_range=0.1,
        shear_range=0.1,
        zoom_range=0.1,
        fill_mode='nearest',
        validation_split=0.2
        ) 

数据加载代码 #

# Loading Training DS
def load_ttv_ds(train_dir,test_dir,val_dir):
  train_ds = datagen.flow_from_directory(
      train_dir,
      target_size=(IMAGE_SIZE, IMAGE_SIZE),
      batch_size=BATCH_SIZE,
      shuffle=True,
      # subset='training'
  )
  test_ds = datagen.flow_from_directory(
      test_dir,
      target_size=(IMAGE_SIZE, IMAGE_SIZE),
      batch_size=BATCH_SIZE,
      shuffle=False
  )
  val_ds = datagen.flow_from_directory(
      val_dir,
      target_size=(IMAGE_SIZE, IMAGE_SIZE),
      batch_size=BATCH_SIZE,
      shuffle=False,
    #   subset="validation"
  )

  return train_ds,test_ds,val_ds

模型构建代码 #

from tensorflow.keras import layers, models
from tensorflow.keras.applications import VGG16

def build_model(lr=0.0001):
    base_model = VGG16(
        weights='imagenet',
        include_top=False,
        input_shape=(256, 256, 3)
    )

    # Freeze the VGG16 layers
    base_model.trainable = False

    model = models.Sequential()
    model.add(base_model)

    model.add(layers.Conv2D(64, kernel_size=3, activation='relu', padding='same'))
    model.add(layers.MaxPooling2D(pool_size=(2, 2)))

    model.add(layers.Conv2D(128, kernel_size=3, activation='relu', padding='same'))
    model.add(layers.MaxPooling2D(pool_size=(2, 2)))

    model.add(layers.Conv2D(256, kernel_size=3, activation='relu', padding='same'))
    model.add(layers.MaxPooling2D(pool_size=(2, 2)))

    model.add(layers.Flatten())

    model.add(layers.Dense(512, activation='relu'))
    model.add(layers.Dropout(0.5))

    model.add(layers.Dense(len(class_names), activation='softmax'))

    model.compile(
        optimizer=keras.optimizers.Adam(learning_rate=lr),
        loss='categorical_crossentropy',
        metrics=['accuracy']
    )

    model.summary()
    return model

训练情况 #

训练25个epoch，训练集准确率始终在0.05-0.08区间波动，测试集准确率为0.05，部分训练日志：

29/29 [==============================] - 36s 1s/step - loss: 5238.8730 - accuracy: 0.0556 - val_loss: 4.9995 - val_accuracy: 0.1211
Epoch 2/25
29/29 [==============================] - 35s 1s/step - loss: 7077.9985 - accuracy: 0.0774 - val_loss: 6.2652 - val_accuracy: 0.0825
Epoch 3/25
29/29 [==============================] - 34s 1s/step - loss: 7518.0254 - accuracy: 0.0720 - val_loss: 5.7178 - val_accuracy: 0.1035

问题根源分析 #

1. 特征过度压缩，丢失关键信息
   VGG16的include_top=False输出特征图尺寸为8×8×512（输入256×256时），之后叠加的3组Conv+MaxPooling将特征压缩至1×1×256，几乎丢失所有空间特征，模型无法学习叶片纹理、形状等判别信息。

2. 未适配VGG16预训练预处理标准
   VGG16基于ImageNet训练，要求输入图像归一化并减去ImageNet均值，当前代码未做对应处理，导致预训练特征无法适配当前任务。

3. 模型结构冗余引发梯度消失
   在预训练特征之上叠加过多卷积层，反向传播时梯度被稀释，底层预训练特征无法有效利用，模型难以收敛。

4. 数据增强误用
   验证集和测试集也应用了数据增强，引入额外噪声，导致评估结果失真，同时干扰模型收敛方向。

5. 训练策略单一
   仅冻结VGG16全部层训练上层网络，自定义网络初始化随机，学习率与训练轮次不匹配，模型无法充分学习任务特征。

针对性解决步骤 #

1. 简化上层网络，避免特征过度压缩 #

移除VGG16后的3组Conv+MaxPooling，改用全局平均池化减少参数与特征损失，同时加入VGG16专属预处理层：

from tensorflow.keras import layers, models
from tensorflow.keras.applications import VGG16
import tensorflow as tf

def build_model(lr=0.0001):
    base_model = VGG16(
        weights='imagenet',
        include_top=False,
        input_shape=(256, 256, 3)
    )
    base_model.trainable = False

    model = models.Sequential()
    # 适配VGG16的预处理：先缩放到[0,1]，再减去ImageNet均值、除以标准差
    model.add(layers.Rescaling(1./255))
    model.add(layers.Normalization(mean=[0.485, 0.456, 0.406], variance=[0.229**2, 0.224**2, 0.225**2]))
    model.add(base_model)
    
    # 全局平均池化替代Flatten，保留空间特征同时减少参数
    model.add(layers.GlobalAveragePooling2D())
    
    model.add(layers.Dense(256, activation='relu'))
    model.add(layers.Dropout(0.5))
    
    model.add(layers.Dense(len(class_names), activation='softmax'))

    model.compile(
        optimizer=tf.keras.optimizers.Adam(learning_rate=lr),
        loss='categorical_crossentropy',
        metrics=['accuracy']
    )
    return model

2. 修正数据增强策略 #

仅训练集使用数据增强，验证集与测试集保持原始图像，避免噪声干扰：

# 训练集数据增强
train_datagen = ImageDataGenerator(
        rotation_range=20,
        width_shift_range=0.1,
        height_shift_range=0.1,
        shear_range=0.1,
        zoom_range=0.1,
        fill_mode='nearest'
        ) 
# 验证集/测试集仅做尺寸调整
val_test_datagen = ImageDataGenerator()

# 修改数据加载函数
def load_ttv_ds(train_dir,test_dir,val_dir):
  train_ds = train_datagen.flow_from_directory(
      train_dir,
      target_size=(IMAGE_SIZE, IMAGE_SIZE),
      batch_size=BATCH_SIZE,
      shuffle=True
  )
  test_ds = val_test_datagen.flow_from_directory(
      test_dir,
      target_size=(IMAGE_SIZE, IMAGE_SIZE),
      batch_size=BATCH_SIZE,
      shuffle=False
  )
  val_ds = val_test_datagen.flow_from_directory(
      val_dir,
      target_size=(IMAGE_SIZE, IMAGE_SIZE),
      batch_size=BATCH_SIZE,
      shuffle=False
  )
  return train_ds,test_ds,val_ds

3. 采用分层迁移训练策略 #

先冻结VGG16全部层训练上层网络，再解冻高层特征层微调，让预训练特征适配当前任务：

# 第一步：训练上层自定义网络
model = build_model(lr=0.0001)
model.fit(train_ds, epochs=10, validation_data=val_ds)

# 第二步：解冻VGG16后4层（共19层Conv+Pooling，高层更贴近语义特征）
base_model = model.layers[2]
base_model.trainable = True
for layer in base_model.layers[:15]:
    layer.trainable = False

# 用更小的学习率重新编译，避免破坏预训练特征
model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-5),
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

# 继续训练30个epoch
model.fit(train_ds, epochs=40, initial_epoch=10, validation_data=val_ds)

4. 平衡类别权重（可选） #

若数据集存在类别样本量差异，计算类别权重平衡训练：

from sklearn.utils.class_weight import compute_class_weight
import numpy as np

class_labels = train_ds.classes
class_weights = compute_class_weight('balanced', classes=np.unique(class_labels), y=class_labels)
class_weight_dict = dict(enumerate(class_weights))

# 训练时传入权重
model.fit(train_ds, epochs=10, validation_data=val_ds, class_weight=class_weight_dict)

内容的提问来源于stack exchange，提问作者Subhodip Roy