如何向TensorFlow Numpy输入函数传入动态长度标签及Estimator验证问题

Let's break down what's happening here first: that ValueError is telling you your training and validation label tensor shapes don't align. Your model expects labels shaped (None, 33) but your validation input is spitting out (None, 37). This almost certainly comes from how you're encoding and splitting your labels—right now you're encoding all labels first, then splitting, which means the training and test sets might end up with different maximum sequence lengths (hence different tensor shapes). And since you don't want to pad labels, we need a way to handle variable-length sequences properly.

Here are two actionable solutions tailored to your goal:

1. Encode Labels with a Unified Length Benchmark (Truncate Instead of Padding) #

If your model is designed to work with fixed-shape dense tensors, you can standardize the label length using the training set's maximum sequence length (never the test set). This way, you only truncate longer labels instead of padding shorter ones (if truncation is acceptable for your OCR task):

# First split your data, then handle label encoding
x_train, x_test, y_train, y_test = dataset_utils.split(features=images, test_size=0.5, labels=labels)

# Calculate max sequence length from training set manually
max_train_seq_len = max(len(label) for label in y_train)

# Encode labels by truncating to the training set's max length
train_encoded = np.array([label[:max_train_seq_len] for label in y_train])
test_encoded = np.array([label[:max_train_seq_len] for label in y_test])

You can also modify your dataset_utils.encode function to support passing a fixed length parameter if you want to keep using that utility instead of manual truncation.

2. Use Sparse Tensors for Variable-Length Labels (No Padding/Truncation) #

If truncation isn't an option, the proper way to handle variable-length labels in TensorFlow Estimators is to use sparse tensors—this lets you pass labels of varying lengths without padding. Here's how to adjust your code:

Step 1: Convert Labels to Sparse Tensors #

Add a helper function to turn your list of variable-length labels into a tf.SparseTensor:

def labels_to_sparse_tensor(labels):
    indices = []
    values = []
    max_seq_len = max(len(label) for label in labels)
    dense_shape = [len(labels), max_seq_len]
    
    for sample_idx, label in enumerate(labels):
        for seq_idx, val in enumerate(label):
            indices.append([sample_idx, seq_idx])
            values.append(val)
    
    return tf.SparseTensor(indices=indices, values=values, dense_shape=dense_shape)

Step 2: Replace numpy_input_fn with Custom tf.data Input Functions #

The built-in numpy_input_fn doesn't handle sparse tensors well, so switch to tf.data for more control:

def train_input_fn():
    # Create dataset from features
    dataset = tf.data.Dataset.from_tensor_slices({
        "x": np.array(x_train),
        "seq_lens": np.array(x_train_seq_lens)
    })
    # Pair features with sparse labels
    dataset = dataset.zip(tf.data.Dataset.from_tensor_slices(y_train))
    dataset = dataset.map(lambda features, label: (features, labels_to_sparse_tensor([label])))
    # Shuffle, batch, and repeat
    dataset = dataset.shuffle(buffer_size=len(x_train)).batch(1).repeat(1)
    return dataset.make_one_shot_iterator().get_next()

def validation_input_fn():
    dataset = tf.data.Dataset.from_tensor_slices({
        "x": np.array(x_test),
        "seq_lens": np.array(x_test_seq_lens)
    })
    dataset = dataset.zip(tf.data.Dataset.from_tensor_slices(y_test))
    dataset = dataset.map(lambda features, label: (features, labels_to_sparse_tensor([label])))
    dataset = dataset.shuffle(buffer_size=len(x_test)).batch(1)
    return dataset.make_one_shot_iterator().get_next()

Step 3: Update Your Model's model_fn to Handle Sparse Labels #

Make sure your model's loss function and metrics can accept sparse tensors. For OCR tasks, tf.nn.ctc_loss is commonly used with sparse labels—adjust your model_fn accordingly, e.g.:

def model_fn(features, labels, mode, params):
    # ... your existing model layer logic ...
    logits = # your model's output logits
    seq_lens = features["seq_lens"]
    
    if mode == tf.estimator.ModeKeys.TRAIN or mode == tf.estimator.ModeKeys.EVAL:
        # Use CTC loss with sparse labels
        loss = tf.nn.ctc_loss(labels=labels, inputs=logits, sequence_length=seq_lens)
        loss = tf.reduce_mean(loss)
        # ... rest of your model_fn logic (optimizers, metrics, etc.) ...

Bonus: Replace Deprecated ValidationMonitor #

Note that learn.monitors.ValidationMonitor is deprecated in newer TensorFlow versions. Instead, use a training loop to run evaluation periodically—it's more flexible and avoids compatibility issues:

num_epochs = 10
steps_per_epoch = len(x_train) // 1  # Match your batch size

for epoch in range(num_epochs):
    print(f"Training epoch {epoch+1}/{num_epochs}")
    classifier.train(input_fn=train_input_fn, steps=steps_per_epoch)
    
    # Run validation after each epoch
    eval_results = classifier.evaluate(input_fn=validation_input_fn)
    print(f"Validation results: {eval_results}")

This achieves the same goal as ValidationMonitor but with better control over evaluation frequency.

内容的提问来源于stack exchange，提问作者Rocket Pingu