ConvLSTM模型用于ASL图像分类的验证损失优化问题咨询

问题描述 #

我正在构建模型用于将ASL（美国手语）图像分类为29个类别，使用的是Kaggle上的美国手语字母数据集。当前遇到的问题是模型验证损失异常飙升，且存在严重过拟合，训练集准确率持续上升，但验证集准确率始终为0。

代码实现 #

from tensorflow.keras.layers import LSTM, MaxPooling3D, Dropout, BatchNormalization, Flatten, Input, Dense, ConvLSTM2D, ConvLSTM1D, TimeDistributed, MaxPooling2D
from tensorflow.keras.models import Sequential
from tensorflow.keras.utils import to_categorical
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from tensorflow.keras.callbacks import EarlyStopping

model = Sequential()
model.add(ConvLSTM2D(1, (5,5), activation = 'tanh', input_shape = (1, 128, 128, 1),
                     recurrent_dropout = 0.2, return_sequences=True, data_format="channels_last",
                     dropout = 0.2, name = "Conv1"))
model.add(MaxPooling3D(pool_size=(1,2,2), padding="same", data_format="channels_last", name = "Pool1"))
model.add(TimeDistributed(Dropout(0.2)))
# model.add(BatchNormalization())

model.add(ConvLSTM2D(5, (3,3), activation = 'tanh', recurrent_dropout = 0.2, return_sequences=True,
                     data_format="channels_last", dropout = 0.2, name = "Conv2"))
model.add(MaxPooling3D(pool_size=(1,2,2), padding="same", data_format="channels_last", name = "Pool2"))
model.add(TimeDistributed(Dropout(0.2)))
# model.add(BatchNormalization())

model.add(ConvLSTM2D(10, (2,3), activation = 'tanh', recurrent_dropout = 0.2, return_sequences=True,
                     data_format="channels_last", dropout = 0.2, name = "Conv3"))
# model.add(MaxPooling3D(pool_size=(1,2,2), padding="same", data_format="channels_last"))
model.add(Flatten())
model.add(Dense(128, activation = "relu", name = "Dense1"))
model.add(Dropout(0.2))
model.add(Dense(29, activation = "softmax", name = "Output_Layer"))

es = EarlyStopping(monitor='val_loss', patience=3, mode = 'min', restore_best_weights=True)
model.compile(optimizer="adam", loss="categorical_crossentropy", metrics=['accuracy'])

r = model.fit(trainX, trainY, epochs=10, shuffle=True, validation_split=0.2, callbacks=[es])

训练输出 #

Epoch 1/10
1088/1088 [==============================] - 132s 107ms/step - loss: 2.8391 - accuracy: 0.1546 - val_loss: 25.4988 - val_accuracy: 0.0000e+00
Epoch 2/10
1088/1088 [==============================] - 115s 106ms/step - loss: 1.6435 - accuracy: 0.5003 - val_loss: 36.5423 - val_accuracy: 0.0000e+00
Epoch 3/10
1088/1088 [==============================] - 178s 164ms/step - loss: 1.1848 - accuracy: 0.6345 - val_loss: 39.9648 - val_accuracy: 0.0000e+00
Epoch 4/10
1088/1088 [==============================] - 124s 114ms/step - loss: 0.8638 - accuracy: 0.7272 - val_loss: 63.1288 - val_accuracy: 0.0000e+00

数据情况 #

原始数据X形状为(43500, 128, 128)，已调整为模型输入形状(43500, 1, 128, 128, 1)；Y形状为(43500, 29)。

问题分析与优化方案 #

核心问题诊断 #

1. 模型结构选型错误：ConvLSTM专为处理序列帧数据（如视频连续帧）设计，但输入时间步仅为1（单张图像），完全无法发挥其循环记忆能力，反而引入冗余循环参数与正则机制，导致训练不稳定、验证集失效。
2. 激活与池化层搭配不合理：tanh激活易引发梯度消失，单时间步下使用MaxPooling3D属于资源浪费，徒增计算开销。
3. 数据未做归一化：原始图像像素值（0-255）未归一化，会导致模型训练时梯度波动剧烈，加剧验证损失飙升。
4. 验证集类别分布可能失衡：未做分层抽样的train_test_split可能导致验证集类别分布异常，直接表现为验证准确率为0。

具体优化步骤 #

1. 替换为普通CNN模型：单图像分类无需序列模型，用Conv2D替代ConvLSTM2D，去除时间步冗余：

   from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dropout, BatchNormalization, Flatten, Dense
   from tensorflow.keras.models import Sequential
   from tensorflow.keras.utils import to_categorical
   from sklearn.preprocessing import LabelEncoder
   from sklearn.model_selection import train_test_split
   from tensorflow.keras.callbacks import EarlyStopping
   
   # 移除冗余时间步，还原输入形状为(43500, 128, 128, 1)
   trainX = trainX.reshape(43500, 128, 128, 1)
   # 像素值归一化到[0,1]
   trainX = trainX / 255.0
   
   # 构建CNN模型
   model = Sequential()
   model.add(Conv2D(16, (5,5), activation='relu', input_shape=(128, 128, 1), padding='same'))
   model.add(MaxPooling2D(pool_size=(2,2)))
   model.add(BatchNormalization())
   model.add(Dropout(0.3))
   
   model.add(Conv2D(32, (3,3), activation='relu', padding='same'))
   model.add(MaxPooling2D(pool_size=(2,2)))
   model.add(BatchNormalization())
   model.add(Dropout(0.3))
   
   model.add(Conv2D(64, (2,2), activation='relu', padding='same'))
   model.add(MaxPooling2D(pool_size=(2,2)))
   model.add(BatchNormalization())
   model.add(Dropout(0.3))
   
   model.add(Flatten())
   model.add(Dense(128, activation='relu', kernel_regularizer='l2'))
   model.add(Dropout(0.3))
   model.add(Dense(29, activation='softmax'))
   
   es = EarlyStopping(monitor='val_loss', patience=5, mode='min', restore_best_weights=True)
   model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
   
   # 分层抽样划分验证集，保证类别分布一致
   X_train, X_val, y_train, y_val = train_test_split(trainX, trainY, test_size=0.2, stratify=trainY, random_state=42)
   
   r = model.fit(X_train, y_train, epochs=20, shuffle=True, validation_data=(X_val, y_val), callbacks=[es])
   

2. 数据标准化：将图像像素值除以255归一化到[0,1]，这是图像分类的标准预处理步骤，可大幅提升训练稳定性。
3. 强化正则与归一化：启用BatchNormalization加速收敛，提高Dropout比例至0.3，给全连接层添加L2正则抑制过拟合。
4. 验证集分布校验：用stratify=trainY做分层抽样，打印y_val的类别分布，确认标签编码无错误。
5. 调整学习率：若训练仍不稳定，可降低Adam学习率至1e-4，避免模型训练震荡。

内容的提问来源于stack exchange，提问作者Dev Jhawar