基于Keras VGG-16的蔬菜图像分类准确率低下问题咨询

我是数据科学领域的新手，此前仅练习过TensorFlow官方示例，这是首次独立完成项目。我正使用Keras、TensorFlow构建图像分类器，数据集包含4种蔬菜（Bell Pepper、Chile Pepper、New Mexico Green Chile、Tomato），每种对应5个子类（damaged、dried、old、ripe、unripe），其中Tomato的dried类无图像，最终共19个类别（已将子类合并为一级文件夹，如Bell_Pepper_Damaged等）。

项目代码 #

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import os
import tensorflow as tf
from tensorflow import keras
from tensorflow import data
from tensorflow.keras import layers
from pathlib import Path
from google.colab import drive

drive.mount('/content/drive')
data_dir = Path('drive/My Drive/VegNet_wo_subclasses')

# Check how many files are included in origin dataset
img_amount = len(list(data_dir.glob('*/*.jpg'))) # '*/*.jpg' indicates all included subfolder images
sub_amount = len(list(data_dir.glob('*'))) # '*' indicates all included subfolders
print('Found {} images in {} subfolders'.format(img_amount, sub_amount))

# Image size recommendation
IMAGE_SIZE = (256, 256)

BATCH_SIZE = 16

VALIDATION_SPLIT = 0.2

SEED = 404

# Split into datasets
train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    data_dir,
    labels = 'inferred',
    label_mode = 'categorical',
    color_mode = 'rgb',
    batch_size = BATCH_SIZE,
    image_size = IMAGE_SIZE,
    shuffle = True,
    seed = SEED,
    validation_split = VALIDATION_SPLIT,
    subset = 'training'
)

class_amount = len(train_ds.class_names)

val_ds = tf.keras.preprocessing.image_dataset_from_directory(
    data_dir,
    labels = 'inferred',
    label_mode = 'categorical',
    color_mode = 'rgb',
    batch_size = BATCH_SIZE,
    image_size = IMAGE_SIZE,
    shuffle = True,
    seed = SEED,
    validation_split = VALIDATION_SPLIT,
    subset = 'validation'
)

# Prefetching samples in GPU memory helps maximize GPU utilization.
train_ds = train_ds.prefetch(tf.data.AUTOTUNE)
val_ds = val_ds.prefetch(tf.data.AUTOTUNE)

# Build model
vgg = keras.applications.VGG16(
    weights="imagenet",
    input_shape=(256, 256, 3),
    include_top=False,
)

vgg.trainable = False

def build_model():

    # create an input mode
    inputs = keras.Input(shape=(256, 256, 3), name="Input")
    
    # add a Flatten or a GlobalAveragePooling layer
    x = layers.Flatten()(vgg.output)

    # add a Dense layer
    x = layers.Dense(19, activation='relu')(x)

    # add a Dropout layer
    x = layers.Dropout(0.2, input_shape = (2,))(x)

    # add the final layer
    outputs = layers.Dense(19)(x)
    
    # build the model
    model = keras.Model(inputs=vgg.input, outputs=outputs)

    # compile the model
    model.compile(loss=keras.losses.BinaryCrossentropy(from_logits=True),
                  optimizer='adam',
                  metrics='accuracy')
    # print the summary
    model.summary()

    return model

model = build_model()

# train the model
history = model.fit(train_ds, 
                    epochs = 20, 
                    validation_data=val_ds,
                    verbose = 1)

模型训练输出 #

Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 256, 256, 3)]     0         
                                                                 
 block1_conv1 (Conv2D)       (None, 256, 256, 64)      1792      
                                                                 
 block1_conv2 (Conv2D)       (None, 256, 256, 64)      36928     
                                                                 
 block1_pool (MaxPooling2D)  (None, 128, 128, 64)      0         
                                                                 
 block2_conv1 (Conv2D)       (None, 128, 128, 128)     73856     
                                                                 
 block2_conv2 (Conv2D)       (None, 128, 128, 128)     147584    
                                                                 
 block2_pool (MaxPooling2D)  (None, 64, 64, 128)       0         
                                                                 
 block3_conv1 (Conv2D)       (None, 64, 64, 256)       295168    
                                                                 
 block3_conv2 (Conv2D)       (None, 64, 64, 256)       590080    
                                                                 
 block3_conv3 (Conv2D)       (None, 64, 64, 256)       590080    
                                                                 
 block3_pool (MaxPooling2D)  (None, 32, 32, 256)       0         
                                                                 
 block4_conv1 (Conv2D)       (None, 32, 32, 512)       1180160   
                                                                 
 block4_conv2 (Conv2D)       (None, 32, 32, 512)       2359808   
                                                                 
 block4_conv3 (Conv2D)       (None, 32, 32, 512)       2359808   
                                                                 
 block4_pool (MaxPooling2D)  (None, 16, 16, 512)       0         
                                                                 
 block5_conv1 (Conv2D)       (None, 16, 16, 512)       2359808   
                                                                 
 block5_conv2 (Conv2D)       (None, 16, 16, 512)       2359808   
                                                                 
 block5_conv3 (Conv2D)       (None, 16, 16, 512)       2359808   
                                                                 
 block5_pool (MaxPooling2D)  (None, 8, 8, 512)         0         
                                                                 
 flatten (Flatten)           (None, 32768)             0         
                                                                 
 dense (Dense)               (None, 19)                622611    
                                                                 
 dropout (Dropout)           (None, 19)                0         
                                                                 
 dense_1 (Dense)             (None, 19)                380       
                                                                 
=================================================================
Total params: 15,337,679
Trainable params: 622,991
Non-trainable params: 14,714,688
_________________________________________________________________
Epoch 1/20
309/309 [==============================] - 338s 953ms/step - loss: 0.6579 - accuracy: 0.1976 - val_loss: 0.5727 - val_accuracy: 0.2006
Epoch 2/20
309/309 [==============================] - 6s 20ms/step - loss: 0.5253 - accuracy: 0.1994 - val_loss: 0.4805 - val_accuracy: 0.2006
Epoch 3/20
309/309 [==============================] - 6s 20ms/step - loss: 0.4441 - accuracy: 0.1994 - val_loss: 0.4108 - val_accuracy: 0.2006
Epoch 4/20
309/309 [==============================] - 6s 20ms/step - loss: 0.3834 - accuracy: 0.1994 - val_loss: 0.3582 - val_accuracy: 0.2006
Epoch 5/20
309/309 [==============================] - 6s 20ms/step - loss: 0.3375 - accuracy: 0.1994 - val_loss: 0.3184 - val_accuracy: 0.2006
Epoch 6/20
309/309 [==============================] - 6s 20ms/step - loss: 0.3027 - accuracy: 0.1994 - val_loss: 0.2882 - val_accuracy: 0.2006
Epoch 7/20
309/309 [==============================] - ETA: 0s - loss: 0.2762 - accuracy: 0.1994

问题1：数据集类别严重不平衡，删除少数类别还是数据增强更合适？ #

优先选择数据增强扩充少数类别，删除类别会直接丢失该类的特征信息，除非该类别完全没有业务价值。具体做法：

• 仅对样本量少的类别应用数据增强，比如随机水平/垂直翻转、小角度旋转、缩放裁剪、亮度/对比度微调、高斯噪声添加等，避免对多数类过度增强导致过拟合。
• 训练时设置class_weight参数，给少数类分配更高的权重，让模型在计算损失时更关注这些类的预测结果。可以用sklearn.utils.class_weight.compute_class_weight自动计算权重。
• 如果少数类样本极少（比如27张），可以考虑采用过采样（重复少数类样本）结合数据增强的方式，不过要注意控制增强后的样本量，避免过拟合。

问题2：基于VGG-16的模型构建是否正确？如何改进？ #

当前模型存在几个关键问题，改进方向如下：

1. 损失函数错误：这是19类的多分类任务，不能用BinaryCrossentropy，应该改用CategoricalCrossentropy(from_logits=True)（因为最后一层没有激活函数），或者给最后一层加softmax激活后用CategoricalCrossentropy()。
2. 分类头设计不合理：
   • 把Flatten换成GlobalAveragePooling2D，减少参数量，避免过拟合，VGG16的输出是(8,8,512)，全局平均池化后得到(512,)的特征，比Flatten后的32768维特征更简洁。
   • 中间的Dense层神经元数量太少，仅19个无法充分学习VGG提取的复杂特征，建议改成256或512个神经元，再搭配Dropout防止过拟合。
   • Dropout的input_shape=(2,)参数完全多余，直接写成layers.Dropout(0.2)即可。
   • 最后一层如果用softmax激活，输出概率分布更符合多分类任务的预期。
3. 模型微调：当前VGG16完全冻结，可以先训练完分类头后，解冻VGG的最后3-5层（比如block5的卷积层），用更小的学习率（比如1e-5）继续训练，让预训练特征适配你的数据集。

改进后的模型示例：

def build_model():
    vgg.trainable = False  # 先冻结全部预训练层
    x = layers.GlobalAveragePooling2D()(vgg.output)
    x = layers.Dense(512, activation='relu')(x)
    x = layers.Dropout(0.3)(x)
    outputs = layers.Dense(19, activation='softmax')(x)
    
    model = keras.Model(inputs=vgg.input, outputs=outputs)
    model.compile(loss=keras.losses.CategoricalCrossentropy(),
                  optimizer=keras.optimizers.Adam(learning_rate=1e-4),
                  metrics='accuracy')
    model.summary()
    return model

# 训练完分类头后解冻顶层微调
model = build_model()
history = model.fit(train_ds, epochs=10, validation_data=val_ds)

# 解冻VGG顶层
vgg.trainable = True
fine_tune_at = len(vgg.layers) - 5  # 解冻最后5层
for layer in vgg.layers[:fine_tune_at]:
    layer.trainable = False

# 重新编译，用更小的学习率
model.compile(loss=keras.losses.CategoricalCrossentropy(),
              optimizer=keras.optimizers.Adam(learning_rate=1e-5),
              metrics='accuracy')
history_fine = model.fit(train_ds, epochs=20, initial_epoch=history.epoch[-1], validation_data=val_ds)

问题3：模型准确率仅约0.1994，如何提升？ #

结合前面的问题，按以下步骤逐步优化：

1. 修正模型核心错误：先把损失函数、分类头的问题解决，这是准确率极低的主要原因之一（当前模型相当于用二分类损失做多分类，完全不匹配任务）。
2. 处理类别不平衡：按照问题1的方法，给少数类做数据增强+类别权重，让模型能学习到少数类的特征。
3. 加入数据增强：不仅针对少数类，对整个训练集也可以加入基础的增强，比如随机翻转、缩放，提升模型的泛化能力。可以用tf.keras.layers.RandomFlip、RandomRotation等层构建数据增强管道：

   data_augmentation = keras.Sequential([
       layers.RandomFlip("horizontal"),
       layers.RandomRotation(0.1),
       layers.RandomZoom(0.1)
   ])
   # 训练时加入增强
   train_ds = train_ds.map(lambda x, y: (data_augmentation(x, training=True), y), num_parallel_calls=tf.data.AUTOTUNE)
   

4. 模型微调：如问题2所述，解冻预训练模型的顶层，用小学习率微调，让预训练特征适配你的蔬菜数据集。
5. 调整训练参数：比如增大batch size到32或64（如果GPU内存允许），调整学习率，增加训练轮次，或者尝试用其他优化器如SGD（带动量）。
6. 分析错误样本：训练后，找出模型分类错误的样本，看看是哪些类别容易混淆（比如不同蔬菜的同状态子类），针对性地补充数据或调整增强策略。

内容的提问来源于stack exchange，提问作者Nick