使用Keras中SimpleRNN时MSE值不匹配问题排查

数据集说明 #

input_train = [[[0], [1], [1], [0], [0], [0], [0], [1], [0], [1]]]
output_train = [[0, 1, 2, 2, 2, 2, 2, 3, 3, 4]]

其中output_train[t]（t取值0-9）等于input_train从i=0到i=t的累加和，用于测试SimpleRNN的工作机制。

测试代码 #

modelo = Sequential()
modelo.add(SimpleRNN(1, input_shape=(number_data_in_simulation,1), activation='linear', return_sequences = True))
modelo.add(Dense(units = 1, activation='linear'))
modelo.compile(optimizer=tf.keras.optimizers.Adam(0.1), loss='mse')
historial = modelo.fit(input_train, output_train, batch_size = 1, epochs=1, verbose=1)

predictions = modelo.predict(input_train, batch_size = 1, verbose = False)
mse_value = 0
number_cases = 0
for sim in range(len(output_train)): 
    for example in range(len(output_train[sim])): 
        mse_value = mse_value + (output_train[sim][example] - predictions[sim][example][0])**2
        number_cases += 1
print(mse_value/number_cases)

终端输出 #

2022-12-15 19:07:11.483935: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2022-12-15 19:07:11.483963: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (jose-HP-EliteBook-830-G7-Notebook-PC): /proc/driver/nvidia/version does not exist
2022-12-15 19:07:11.484223: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
1/1 [==============================] - 0s 467ms/step - loss: 6.1238
5.149037324060892

预期训练输出的loss与手动计算的MSE一致，但两者数值不相等，求解原因。

核心原因 #

Keras的fit函数输出的loss是参数更新前的初始模型预测值计算的MSE，而手动计算的是参数更新后模型的预测值对应的MSE。因为设置了epochs=1，模型在这一轮训练中完成了一次参数更新，两个阶段的模型参数不同，计算出的MSE自然有差异。

验证方式 #

在fit执行前，先计算初始模型的MSE，这个值会和fit输出的6.1238几乎一致：

# 在model.compile之后、model.fit之前添加以下代码
initial_predictions = modelo.predict(input_train, batch_size=1, verbose=False)
initial_mse = 0
count = 0
for sim in range(len(output_train)):
    for example in range(len(output_train[sim])):
        initial_mse += (output_train[sim][example] - initial_predictions[sim][example][0])**2
        count += 1
print("初始模型MSE:", initial_mse/count)

额外优化建议 #

• 模型结构冗余：SimpleRNN已经输出维度为1的序列，后续的Dense(1)层仅做线性变换，无实际意义，建议直接移除以简化结构。
• 学习率设置过大：Adam优化器的学习率设为0.1容易导致训练不稳定，建议调整为0.001或0.01。

内容的提问来源于stack exchange，提问作者Jose Gonzalez B