TensorFlow 1.0 RNN训练损失停滞问题排查求助

因兼容性限制无法切换到TensorFlow 2.0，目前用TensorFlow 1.0编写简易RNN时遇到异常：前4次迭代损失有变化，之后就停滞在固定值。调整学习率等参数都没改善，模型效果极差。推测代码有错误，但对TensorFlow 1.0不熟悉，无法定位问题，请求帮忙排查。

复现代码 #

import numpy as np

import matplotlib.pyplot as plt
from matplotlib import rcParams
import matplotlib.cm as cm

import tensorflow as tf
tf.set_random_seed(5)  # 5

import sys
sys.path.append("..")

# ===================================================================================
#                                   Hyperparameters
# ===================================================================================

batch_size = 300

# number of optimization steps
reps = 100 

lr = 0.01 
b1 = 0.9
b2 = 0.999
eps = 1e-08 

# ===================================================================================
#                                   Training data
# ===================================================================================
# load the training data from the provided files

train_data = np.load('./inputs_train.npy') 
test_data = np.load('./inputs_test.npy')
train_y = np.load('./labels_train.npy').reshape(batch_size, 1) 
test_y = np.load('./labels_test.npy').reshape(batch_size, 1) 

########################### Classical NN ################################

d = 4 # number of predictors
p = 16 # best: 16
input_neurons = d
output_neurons = 1

# initialise hidden state

h = tf.placeholder(tf.float32, shape=[batch_size, p])
hid = np.ones((batch_size, p))

input_layer = tf.placeholder(tf.float32, shape=[batch_size, input_neurons])

input_matrix = tf.Variable(tf.random_normal(shape=[input_neurons, p]))
offset_input = tf.Variable(tf.random_normal(shape=[p]))

hidden_matrix = tf.Variable(tf.random_normal(shape=[p, p]))
offset_hidden = tf.Variable(tf.random_normal(shape=[p]))

output_matrix = tf.Variable(tf.random_normal(shape=[p, output_neurons]))
offset_output = tf.Variable(tf.random_normal(shape=[output_neurons]))

h_t = tf.nn.elu(tf.matmul(input_layer, input_matrix)+offset_input + tf.matmul(h, hidden_matrix)+offset_hidden)
output_layer = tf.nn.elu(tf.matmul(h_t, output_matrix) + offset_output)

parameters = [input_matrix, offset_input, hidden_matrix, offset_hidden, output_matrix, offset_output]

# ===================================================================================
#                      Define the loss function
# ===================================================================================

hidden = h_t
pred = output_layer

output_data = tf.placeholder(tf.float32, shape=[batch_size, 1])
loss = tf.reduce_mean(tf.abs(pred-output_data)**2) 

# ===================================================================================
#                      Perform the optimization
# ===================================================================================

# we choose the Adam optimizer
optimiser = tf.train.AdamOptimizer(learning_rate=lr,
                                    beta1=b1,
                                    beta2=b2,
                                    epsilon=eps)
min_op = optimiser.minimize(loss)

session = tf.Session()
session.run(tf.global_variables_initializer())

print('Beginning optimization')

loss_vals = []

for i in range(reps+1):

    loss_, predictions, hid, _ = session.run(
        [loss, pred, hidden, min_op],
        feed_dict={input_layer: train_data, h: hid, output_data: train_y})
    loss_vals.append(loss_)
        
    print('Step: {} Loss: {}'.format(i, loss_))

损失输出 #

Beginning optimization
Step: 0 Loss: 1.1433227062225342
Step: 1 Loss: 1.2459838390350342
Step: 2 Loss: 1.245898723602295
Step: 3 Loss: 1.2459838390350342
Step: 4 Loss: 1.2459838390350342
Step: 5 Loss: 1.2459838390350342
Step: 6 Loss: 1.2459838390350342
Step: 7 Loss: 1.2459838390350342
Step: 8 Loss: 1.2459838390350342
Step: 9 Loss: 1.2459838390350342
Step: 10 Loss: 1.2459838390350342
Step: 11 Loss: 1.2459838390350342
Step: 12 Loss: 1.2459838390350342
Step: 13 Loss: 1.2459838390350342
Step: 14 Loss: 1.2459838390350342
Step: 15 Loss: 1.2459838390350342
Step: 16 Loss: 1.2459838390350342
Step: 17 Loss: 1.2459838390350342
Step: 18 Loss: 1.2459838390350342
Step: 19 Loss: 1.2459838390350342
Step: 20 Loss: 1.2459838390350342
...

问题定位 #

• RNN核心逻辑缺失：当前代码仅实现了单步隐藏层计算，没有按序列迭代传递隐藏状态。每次训练都使用初始的全1隐藏状态hid = np.ones((batch_size, p))，本质是普通前馈网络，而非循环神经网络，无法学习时序依赖。
• 输入维度不符合RNN要求：RNN需要处理[batch_size, sequence_length, input_dim]格式的序列数据，但当前input_layer维度是[batch_size, input_neurons]，没有体现序列结构，模型无法捕捉时序信息。
• 输出激活函数选择错误：回归任务最后一层用ELU会限制输出范围（x<0时输出为e^x-1，x>0时等于x），若标签包含负值，模型无法拟合，直接导致损失停滞。
• 训练循环逻辑错误：每次迭代都喂入完整的train_data，没有按序列步长拆分输入，也未在序列步骤间更新传递隐藏状态，模型无法学习时序规律。

修改建议 #

1. 重构RNN序列处理逻辑：将输入数据调整为[batch_size, seq_len, input_dim]的序列格式，在训练循环中遍历每个时间步，传递更新后的隐藏状态。
2. 调整输出层激活函数：回归任务输出层改用线性激活，去掉ELU：

   output_layer = tf.matmul(h_t, output_matrix) + offset_output
   

3. 修正隐藏状态传递：如果是单步时序任务，确保每次迭代后用更新的hid覆盖初始值；如果是多步序列，需在循环内按时间步逐步计算并传递隐藏状态。
4. 检查数据维度：确认train_data是否为时序数据，若原始数据是单步样本，需重新设计模型结构；若为时序数据，调整维度后再训练。

内容的提问来源于stack exchange，提问作者Donna Schweitzer