使用TFLearn Trainer训练双向RNN时触发RecursionError

I’ve run into similar headaches before when combining TFLearn’s high-level Trainer with custom TensorFlow ops like CTC loss and sparse placeholders. That RecursionError almost always comes from TFLearn’s internal tensor dependency tracking getting stuck in an infinite loop when dealing with dynamic RNN outputs or sparse tensors used in CTC calculations.

Here are two actionable fixes you can try:

Option 1: Replace TFLearn Trainer with a Native TensorFlow Training Loop #

TFLearn’s Trainer isn’t always the best fit for highly custom models (especially those using CTC or sparse inputs). Switching to a raw TensorFlow loop avoids the recursive tracking issues entirely:

import dataset_utils
import tensorflow as tf
import tflearn
from tensorflow.contrib import grid_rnn

def main(_):
    image_paths, labels = dataset_utils.read_dataset_list('../test/dummy_labels_file.txt')
    data_dir = "../test/dummy_data/"
    images = dataset_utils.read_images(data_dir=data_dir, image_paths=image_paths, image_extension='png')
    print('Done reading images')
    images = dataset_utils.resize(images, (1596, 48))
    images = dataset_utils.transpose(images)
    labels = dataset_utils.encode(labels)
    x_train, x_test, y_train, y_test = dataset_utils.split(features=images, test_size=0.5, labels=labels)
    y_train = dataset_utils.convert_to_sparse(y_train)
    y_test = dataset_utils.convert_to_sparse(y_test)

    with tf.Graph().as_default():
        X = tf.placeholder(tf.float32, [None, None, 48])
        Y = tf.sparse_placeholder(tf.int32)
        seq_lens = tf.placeholder(tf.int32, [None])

        def dnn(x):
            cell_fw = grid_rnn.Grid2LSTMCell(num_units=128)
            cell_bw = grid_rnn.Grid2LSTMCell(num_units=128)
            bidirectional_grid_rnn, _ = tf.nn.bidirectional_dynamic_rnn(cell_fw, cell_bw, x, dtype=tf.float32)
            outputs = tf.concat(bidirectional_grid_rnn, axis=-1)
            outputs = tf.reshape(outputs, [-1, 256])
            W = tf.Variable(tf.truncated_normal([256, 80], stddev=0.1, dtype=tf.float32), name='W')
            b = tf.Variable(tf.constant(0., dtype=tf.float32, shape=[80]), name='b')
            logits = tf.matmul(outputs, W) + b
            logits = tf.reshape(logits, [tf.shape(x)[0], -1, 80])
            logits = tf.transpose(logits, (1, 0, 2))
            return logits

        net = dnn(X)
        decoded, _ = tf.nn.ctc_beam_search_decoder(net, seq_lens, merge_repeated=False)
        cost = tf.reduce_mean(tf.nn.ctc_loss(inputs=net, labels=Y, sequence_length=seq_lens))
        optimizer = tf.train.MomentumOptimizer(learning_rate=0.001, momentum=0.5)
        train_op = optimizer.minimize(cost)
        label_error_rate = tf.reduce_mean(tf.edit_distance(tf.cast(decoded[0], tf.int32), Y))

        # Native TensorFlow training loop
        with tf.Session() as sess:
            sess.run(tf.global_variables_initializer())
            
            # Training step
            train_feed = {X: x_train, Y: y_train, seq_lens: dataset_utils.get_seq_lens(x_train)}
            _, train_loss, train_ler = sess.run([train_op, cost, label_error_rate], feed_dict=train_feed)
            print(f"Training Loss: {train_loss}, Label Error Rate: {train_ler}")
            
            # Validation step
            val_feed = {X: x_test, Y: y_test, seq_lens: dataset_utils.get_seq_lens(x_test)}
            val_loss, val_ler = sess.run([cost, label_error_rate], feed_dict=val_feed)
            print(f"Validation Loss: {val_loss}, Label Error Rate: {val_ler}")

if __name__ == '__main__':
    tf.app.run(main=main)

Key changes here:

• Removed tflearn.TrainOp and tflearn.Trainer entirely
• Explicitly define the training operation with optimizer.minimize(cost)
• Use a native TensorFlow Session to run training and validation steps manually

Option 2: Adjust TFLearn Trainer Setup to Break Recursion #

If you need to keep using TFLearn’s Trainer, you can try wrapping your loss and metric operations to break the recursive dependency chain. The issue often arises when TFLearn tries to track dependencies for sparse tensors used in CTC loss.

Modify your TrainOp initialization like this:

# Wrap cost and metric to avoid direct sparse tensor tracking
cost = tf.reduce_mean(tf.nn.ctc_loss(inputs=net, labels=Y, sequence_length=seq_lens))
cost = tf.identity(cost, name="ctc_cost")  # Add identity to break dependency chain

label_error_rate = tf.reduce_mean(tf.edit_distance(tf.cast(decoded[0], tf.int32), Y))
label_error_rate = tf.identity(label_error_rate, name="ler_metric")

train_op = tflearn.TrainOp(loss=cost, optimizer=optimizer, metric=label_error_rate, batch_size=1)
trainer = tflearn.Trainer(train_ops=train_op, tensorboard_verbose=0)

Adding tf.identity creates a new tensor that breaks the recursive tracking loop TFLearn might be stuck in. Note that this isn’t guaranteed to work for all complex cases, but it’s worth a shot if you want to keep using TFLearn’s utilities.

Why This Happens #

TFLearn’s Trainer uses recursive tensor traversal to build dependency graphs for automatic differentiation and training. When working with sparse tensors (like your CTC labels) or dynamic RNN outputs with variable shapes, this traversal can accidentally enter an infinite recursion loop, triggering the RecursionError.

内容的提问来源于stack exchange，提问作者Rocket Pingu