LSTM手语分类模型未学习，验证准确率波动异常求助

问题背景 #

针对3种简单手语（No sign/Yes/No）构建分类器：通过MediaPipe提取人脸、手部、姿态关键点，堆叠为1662维数组保存为.npy文件；每个手语对应30帧时序数据，共540个训练样本、60个验证样本。采用LSTM模型训练，但训练与验证准确率波动异常，模型未有效学习。

代码实现 #

Data_Path_train = "training_data"
Data_Path_val = "validation_data"
actions = np.array(['No sign', 'Yes', 'No'])

label_map = {label:num for num, label in enumerate(actions)}
print(label_map)

def data_generator(data_path, actions, sequence_length, batch_size,validation):
    while True:
        for action in actions:
            for sequence in os.listdir(os.path.join(data_path, action)):
                # skip non-numeric sequence names (i.e., augmented sequences)
                if not sequence.isdigit():
                    continue
                if validation and int(sequence) % 5 == 0:
                    continue
                if not validation and int(sequence) % 5 != 0:
                    continue
                sequence = int(sequence)
                window = []
                for frame_num in range(sequence_length):
                    res = np.load(os.path.join(data_path, action, str(sequence), "{}.npy".format(frame_num)))
                    window.append(res)
                x = np.array(window)
                y = keras.utils.to_categorical(label_map[action], num_classes=len(actions))
                yield x[np.newaxis, :, :], y[np.newaxis, :]

sequence_length = 30
batch_size = 60

train_gen = data_generator(Data_Path_train, actions, sequence_length, batch_size,validation=False)
val_gen = data_generator(Data_Path_val, actions, sequence_length, batch_size,validation=True)

from keras.models import Sequential
from keras.layers import LSTM, Dense

# Define the input shape
input_shape = (sequence_length, 1662)

# Define the model architecture
model = Sequential()
model.add(LSTM(64, input_shape=input_shape, return_sequences=True))
model.add(LSTM(64, return_sequences=False))
model.add(Dense(len(actions), activation='softmax'))


# Compile the model
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
history = model.fit(train_gen, 
                    steps_per_epoch=540*len(actions)//batch_size, 
                    validation_data=val_gen, 
                    validation_steps=60*len(actions)//batch_size, 
                    epochs=30,verbose=1, shuffle=True)

训练结果 #

Epoch 1/30
27/27 [==============================] - 2s 56ms/step - loss: 1.3371 - accuracy: 0.5926 - val_loss: 1.7992 - val_accuracy: 0.3333
Epoch 2/30
27/27 [==============================] - 1s 21ms/step - loss: 2.3097 - accuracy: 0.1852 - val_loss: 2.4106 - val_accuracy: 0.0000e+00
Epoch 3/30
27/27 [==============================] - 1s 21ms/step - loss: 1.6771 - accuracy: 0.0000e+00 - val_loss: 1.3552 - val_accuracy: 0.0000e+00
Epoch 4/30
27/27 [==============================] - 1s 21ms/step - loss: 1.3889 - accuracy: 0.0741 - val_loss: 1.3459 - val_accuracy: 0.0000e+00
Epoch 5/30
27/27 [==============================] - 1s 25ms/step - loss: 1.2858 - accuracy: 0.0000e+00 - val_loss: 1.2403 - val_accuracy: 0.3333
Epoch 6/30
27/27 [==============================] - 1s 27ms/step - loss: 1.2816 - accuracy: 0.0370 - val_loss: 1.0072 - val_accuracy: 0.6667
Epoch 7/30
27/27 [==============================] - 1s 22ms/step - loss: 1.1760 - accuracy: 0.0000e+00 - val_loss: 1.0123 - val_accuracy: 0.0000e+00
Epoch 8/30
27/27 [==============================] - 1s 22ms/step - loss: 1.2672 - accuracy: 0.0370 - val_loss: 1.1520 - val_accuracy: 0.0000e+00
Epoch 9/30
27/27 [==============================] - 1s 22ms/step - loss: 1.1903 - accuracy: 0.0000e+00 - val_loss: 1.2613 - val_accuracy: 0.0000e+00
Epoch 10/30
27/27 [==============================] - 1s 20ms/step - loss: 1.2038 - accuracy: 0.0370 - val_loss: 1.1799 - val_accuracy: 0.3333
Epoch 11/30
27/27 [==============================] - 1s 21ms/step - loss: 1.2164 - accuracy: 0.0000e+00 - val_loss: 1.0480 - val_accuracy: 0.6667
Epoch 12/30
27/27 [==============================] - 1s 21ms/step - loss: 1.1805 - accuracy: 0.0000e+00 - val_loss: 1.0215 - val_accuracy: 1.0000
Epoch 13/30
27/27 [==============================] - 1s 20ms/step - loss: 1.1981 - accuracy: 0.0370 - val_loss: 1.0742 - val_accuracy: 0.0000e+00
Epoch 14/30
27/27 [==============================] - 1s 21ms/step - loss: 1.1769 - accuracy: 0.0000e+00 - val_loss: 1.1262 - val_accuracy: 0.0000e+00
Epoch 15/30
27/27 [==============================] - 1s 20ms/step - loss: 1.1846 - accuracy: 0.0370 - val_loss: 1.1544 - val_accuracy: 0.0000e+00
Epoch 16/30
27/27 [==============================] - 1s 21ms/step - loss: 1.1534 - accuracy: 0.0370 - val_loss: 1.0768 - val_accuracy: 0.6667
Epoch 17/30
27/27 [==============================] - 1s 22ms/step - loss: 1.1903 - accuracy: 0.0370 - val_loss: 0.9775 - val_accuracy: 1.0000
Epoch 18/30
27/27 [==============================] - 1s 21ms/step - loss: 1.1363 - accuracy: 0.0000e+00 - val_loss: 1.0444 - val_accuracy: 0.0000e+00
Epoch 19/30
27/27 [==============================] - 1s 20ms/step - loss: 1.1847 - accuracy: 0.0741 - val_loss: 1.1029 - val_accuracy: 0.0000e+00
Epoch 20/30
27/27 [==============================] - 1s 20ms/step - loss: 1.1653 - accuracy: 0.0000e+00 - val_loss: 1.1406 - val_accuracy: 0.0000e+00
Epoch 21/30
27/27 [==============================] - 1s 20ms/step - loss: 1.1462 - accuracy: 0.0741 - val_loss: 1.0914 - val_accuracy: 0.6667
Epoch 22/30
27/27 [==============================] - 1s 21ms/step - loss: 1.1722 - accuracy: 0.0000e+00 - val_loss: 1.0319 - val_accuracy: 1.0000
Epoch 23/30
27/27 [==============================] - 1s 21ms/step - loss: 1.1600 - accuracy: 0.0370 - val_loss: 1.0185 - val_accuracy: 1.0000
Epoch 24/30
27/27 [==============================] - 1s 21ms/step - loss: 1.1582 - accuracy: 0.0000e+00 - val_loss: 1.0609 - val_accuracy: 1.0000
Epoch 25/30
27/27 [==============================] - 1s 20ms/step - loss: 1.1511 - accuracy: 0.0741 - val_loss: 1.1442 - val_accuracy: 0.0000e+00
Epoch 26/30
27/27 [==============================] - 1s 23ms/step - loss: 1.1811 - accuracy: 0.0000e+00 - val_loss: 1.1506 - val_accuracy: 0.0000e+00
Epoch 27/30
27/27 [==============================] - 1s 21ms/step - loss: 1.1193 - accuracy: 0.1481 - val_loss: 0.9981 - val_accuracy: 1.0000
Epoch 28/30
27/27 [==============================] - 1s 21ms/step - loss: 1.1849 - accuracy: 0.0000e+00 - val_loss: 1.0087 - val_accuracy: 1.0000
Epoch 29/30
27/27 [==============================] - 1s 21ms/step - loss: 1.1146 - accuracy: 0.0000e+00 - val_loss: 1.0342 - val_accuracy: 0.0000e+00
Epoch 30/30
27/27 [==============================] - 1s 23ms/step - loss: 1.1934 - accuracy: 0.1481 - val_loss: 1.0982 - val_accuracy: 0.6667

问题排查与优化建议 #

1. 数据生成器批量逻辑错误 #

当前生成器每次仅返回1个样本（通过x[np.newaxis,:,:]和y[np.newaxis,:]扩展维度），但batch_size设为60，steps_per_epoch按总样本数/60计算，导致模型实际每次step仅处理1个样本，无法学习批量数据分布，且shuffle=True对顺序输出的生成器无效。

• 修复：修改生成器，累积batch_size个样本后再yield；或改用keras.utils.Sequence类实现更规范的批量生成。

2. 训练/验证数据划分重复且混乱 #

已单独划分training_data和validation_data文件夹，但生成器中仍通过sequence%5二次筛选样本，导致实际训练/验证样本数量与预期（540/60）不符，数据分布被破坏。

• 修复：删除生成器中所有validation相关的判断逻辑，直接读取对应文件夹下的全部样本。

3. 模型容量不足，未适配高维时序输入 #

输入为1662维高维数据，仅用两层64单元LSTM，模型容量不足以捕捉手语的动态时序特征；且无正则化层，易出现训练波动。

• 优化：增加LSTM单元数（如128/256）；在LSTM层后添加Dropout(0.2)层抑制过拟合；在输出Dense层前新增一层Dense隐藏层（如Dense(32, activation='relu')）。

4. 数据未归一化 #

MediaPipe提取的关键点坐标（x/y/z）数值范围差异大，未归一化会导致模型训练不稳定，损失和准确率波动剧烈。

• 修复：对每个样本的关键点进行归一化，如缩放到[0,1]或[-1,1]范围；或采用标准化（均值为0，方差为1）处理。

5. 样本类别可能不平衡 #

若三类样本数量差异较大，模型会偏向预测样本多的类别，导致准确率异常波动。

• 修复：统计每个类别的样本数，若不平衡，采用过采样、欠采样或在model.fit中设置class_weight参数调整类别权重。

内容的提问来源于stack exchange，提问作者Dhaval29