如何提升ANN多分类任务的验证准确率？训练精度高但验证精度停滞

问题背景 #

构建人工神经网络解决3分类问题，训练准确率随epoch持续上升（最高达91.61%），但验证准确率始终停滞在0.4~0.5之间，训练损失远低于验证损失，模型严重过拟合。

数据集示例 #

project_type_id_Apartemen   project_type_id_Bandara project_type_id_Dam Weir    project_type_id_Hotel   project_type_id_Industry Petrochemical  project_type_id_Irigasi project_type_id_Jalan Jembatan  project_type_id_Komersil    project_type_id_Lain-lain   project_type_id_Oil Gas ... area_ASA    area_BALI&NUSA TENGGARA area_JAKARTA    area_JAWA   area_KALIMANTAN area_MALUKU area_PAPUA  area_SULAWESI   area_SUMATERA   area_TIMORLESTE
174 0   0   0   0   0   0   0   1   0   0   ... 0   0   0   1   0   0   0   0   0   0
133 1   0   0   0   0   0   0   0   0   0   ... 0   0   1   0   0   0   0   0   0   0
326 0   0   0   0   0   0   0   1   0   0   ... 0   0   0   0   0   0   0   1   0   0
473 0   0   0   0   0   0   0   0   0   0   ... 0   0   0   0   0   0   0   0   1   0
55  0   0   1   0   0   0   0   0   0   0   ... 0   0   0   1   0   0   0   0   0   0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
71  1   0   0   0   0   0   0   0   0   0   ... 0   0   0   0   0   0   0   0   1   0
106 0   0   0   0   0   0   0   0   0   0   ... 0   0   0   0   0   0   0   0   1   0
270 0   0   0   1   0   0   0   0   0   0   ... 0   0   0   0   0   0   0   1   0   0
435 0   0   0   0   0   0   0   0   0   0   ... 0   0   0   0   0   0   0   0   1   0
102 0   0   0   0   0   0   0   0   0   0   ... 0   0   0   0   0   0   0   0   1   0

模型代码 #

model = keras.Sequential([
    keras.layers.Dense(37,input_dim=37,activation='relu'),
    keras.layers.Dropout(0.1),
    keras.layers.Dense(30,activation='relu'),
    keras.layers.Dropout(0.1),
    keras.layers.Dense(20,activation='relu'),
    keras.layers.Dropout(0.1),
    keras.layers.Dense(10,activation='relu'),
    keras.layers.Dropout(0.1),
    keras.layers.Dense(3,activation='softmax'),
])

model.compile(optimizer='Adam',loss='categorical_crossentropy',metrics=['accuracy'])
model.fit(x=X_train,y=y_train,
          validation_data=(X_test,y_test),
          batch_size=128,epochs=2500)

训练输出片段 #

Epoch 844/2500
5/5 [==============================] - 0s 30ms/step - loss: 0.2595 - accuracy: 0.8766 - val_loss: 3.1100 - val_accuracy: 0.4277
Epoch 845/2500
5/5 [==============================] - 0s 17ms/step - loss: 0.2657 - accuracy: 0.8924 - val_loss: 3.1053 - val_accuracy: 0.4214
Epoch 846/2500
5/5 [==============================] - 0s 15ms/step - loss: 0.2840 - accuracy: 0.8671 - val_loss: 3.0908 - val_accuracy: 0.4277
Epoch 847/2500
5/5 [==============================] - 0s 22ms/step - loss: 0.2639 - accuracy: 0.8987 - val_loss: 3.0742 - val_accuracy: 0.4403
Epoch 848/2500
5/5 [==============================] - 0s 14ms/step - loss: 0.3026 - accuracy: 0.8718 - val_loss: 3.0469 - val_accuracy: 0.4403
Epoch 849/2500
5/5 [==============================] - 0s 14ms/step - loss: 0.2802 - accuracy: 0.8845 - val_loss: 3.0113 - val_accuracy: 0.4528
Epoch 850/2500
5/5 [==============================] - 0s 13ms/step - loss: 0.2614 - accuracy: 0.8892 - val_loss: 2.9987 - val_accuracy: 0.4528
Epoch 851/2500
5/5 [==============================] - 0s 13ms/step - loss: 0.2593 - accuracy: 0.8940 - val_loss: 2.9887 - val_accuracy: 0.4403
Epoch 852/2500
5/5 [==============================] - 0s 16ms/step - loss: 0.2260 - accuracy: 0.9161 - val_loss: 3.0022 - val_accuracy: 0.4403
Epoch 853/2500
5/5 [==============================] - 0s 17ms/step - loss: 0.2669 - accuracy: 0.8845 - val_loss: 3.0241 - val_accuracy: 0.4654
Epoch 854/2500
5/5 [==============================] - 0s 52ms/step - loss: 0.2913 - accuracy: 0.8703 - val_loss: 3.0524 - val_accuracy: 0.4277
Epoch 855/2500
5/5 [==============================] - 0s 51ms/step - loss: 0.2261 - accuracy: 0.9019 - val_loss: 3.0725 - val_accuracy: 0.4340
Epoch 856/2500
5/5 [==============================] - 0s 46ms/step - loss: 0.2658 - accuracy: 0.8813 - val_loss: 3.0963 - val_accuracy: 0.4340

解决措施 #

1. 数据集处理 #

• 标准化数值特征：第一列数值特征（如174、133）需用StandardScaler归一化，避免大数值主导模型：

  from sklearn.preprocessing import StandardScaler
  scaler = StandardScaler()
  X_train[:, 0] = scaler.fit_transform(X_train[:, 0].reshape(-1,1)).flatten()
  X_test[:, 0] = scaler.transform(X_test[:, 0].reshape(-1,1)).flatten()
  

• 检查类别平衡：统计训练/验证集三类样本数量，若不平衡，使用class_weight='balanced'参数，或对少数类过采样。
• 验证集划分合理性：确保验证集和训练集的分布一致，避免随机划分导致分布偏移。

2. 模型简化与正则化 #

• 减少模型复杂度：原模型4个隐藏层过于冗余，简化为2层隐藏层即可：

  model = keras.Sequential([
      keras.layers.Dense(20, input_dim=37, activation='relu', kernel_regularizer=keras.regularizers.l2(0.001)),
      keras.layers.Dropout(0.3),
      keras.layers.Dense(10, activation='relu', kernel_regularizer=keras.regularizers.l2(0.001)),
      keras.layers.Dropout(0.3),
      keras.layers.Dense(3, activation='softmax'),
  ])
  

• 提高Dropout比例：将原0.1的Dropout比例提升至0.3，增强正则化效果。

3. 训练策略调整 #

• 添加早停机制：避免过度训练，当验证损失连续20轮不下降时停止，恢复最优权重：

  from tensorflow.keras.callbacks import EarlyStopping
  early_stop = EarlyStopping(monitor='val_loss', patience=20, restore_best_weights=True, verbose=1)
  model.fit(x=X_train,y=y_train,
            validation_data=(X_test,y_test),
            batch_size=64,epochs=2500,
            callbacks=[early_stop])
  

• 降低学习率：将Adam优化器的学习率从默认0.001调整为0.0001，让模型学习更平缓：

  model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.0001),
                loss='categorical_crossentropy',metrics=['accuracy'])
  

• 调整Batch Size：将batch_size从128降至64，小批量更新有助于模型泛化。

内容的提问来源于stack exchange，提问作者Alika Pratama