基于Keras实现百度DeepSpeech1时CTC损失停滞且输出全空白的问题

I'm trying to implement Baidu's DeepSpeech1 using CTC loss in Keras, but I'm hitting a roadblock during training. My loss drops from 1000+ to the 100-200 range and then stops decreasing entirely. When testing (by removing the Lambda layer to get transcription results), the model only outputs blank characters. I suspect the model is stuck in a local minimum where outputting blanks gives lower loss than random characters, but it's not actually learning to transcribe speech.

Here's my code:

import numpy as np
from tensorflow.keras.utils import Sequence
from tensorflow.keras.layers import Input, TimeDistributed, Dense, Activation, Dropout, Bidirectional, LSTM, Lambda
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.activations import relu
from tensorflow.keras.backend import ctc_batch_cost

class dataGen(Sequence): # data generator for Mozilla common voice
    def __init__(self, audiopaths, transcripts, batch_size):
        self.x = audiopaths
        self.y = transcripts
        self.batch_size = batch_size
    def __len__(self):
        return int(len(self.x) / self.batch_size)
    def __getitem__(self, idx):
        batch_x = self.x[idx*self.batch_size : (idx+1)*self.batch_size]
        batch_y = self.y[idx*self.batch_size : (idx+1)*self.batch_size]
        x_val = [get_max_time(file_name) for file_name in batch_x]
        max_val = max(x_val)
        x_data = np.array([make_mfcc_shape(file_name, padlen=max_val) for file_name in batch_x]) # just converts data to mfcc
        y_val = [get_maxseq_len(l) for l in batch_y]
        max_y = max(y_val)
        labels = np.array([get_intseq(l, max_intseq_length=max_y) for l in batch_y])
        input_length = np.array(x_val)
        label_length = np.array(y_val)
        return [x_data, labels, input_length, label_length], np.zeros((self.batch_size,)), [None]
    def on_epoch_end(self):
        i = np.arange(len(self.x))
        np.random.shuffle(i)
        self.x = self.x[i]
        self.y = self.y[I]  # <-- Notice the typo here!

def clipped_relu(x):
    return relu(x, max_value=20)

def ctc_lambda_func(args):
    y_pred, labels, input_length, label_length = args
    return ctc_batch_cost(labels, y_pred, input_length, label_length)

def ctc(y_true, y_pred):
    return y_pred

input_data = Input(name='the_input', shape=(None, 26))
inner = TimeDistributed(Dense(2048))(input_data)
inner = TimeDistributed(Activation(clipped_relu))(inner)
inner = TimeDistributed(Dropout(0.1))(inner)
inner = TimeDistributed(Dense(2048))(inner)
inner = TimeDistributed(Activation(clipped_relu))(inner)
inner = TimeDistributed(Dropout(0.1))(inner)
inner = TimeDistributed(Dense(2048))(inner)
inner = TimeDistributed(Activation(clipped_relu))(inner)
inner = TimeDistributed(Dropout(0.1))(inner)
inner = Bidirectional(LSTM(2048, return_sequences=True))(inner)
inner = TimeDistributed(Activation(clipped_relu))(inner)
inner = TimeDistributed(Dropout(0.1))(inner)
output = TimeDistributed(Dense(28, activation="softmax"))(inner)

labels = Input(name='the_labels', shape=[None,])
input_length = Input(name='input_length', shape=[1])
label_length = Input(name='label_length', shape=[1])

loss_out = Lambda(ctc_lambda_func, output_shape=(1,), name='ctc')([output, labels, input_length, label_length])
model = Model(inputs=[input_data, labels, input_length, label_length], outputs=loss_out)
model.compile(optimizer='adam', loss=ctc)

Let's break down the issues and fixes step by step—your intuition about the local minimum is plausible, but there are several concrete fixes and checks you can apply:

1. Fix the Critical Typo in Data Generator #

First, look at your on_epoch_end method:

self.y = self.y[I]  # <-- Typo here!

You're using a capital I instead of the shuffled index array i. This means your transcript labels aren't being shuffled alongside your audio paths—so after the first epoch, your model is training on mismatched audio-label pairs. That's a showstopper for learning, and explains why it's defaulting to blank outputs. Correct this to:

self.y = self.y[i]

2. Validate CTC Input Dimensions & Label Mapping #

CTC is extremely sensitive to correct input shapes and label formatting:

• Check label shape: Your labels Input is defined as shape=[None,], but ctc_batch_cost expects y_true to be a 2D array (batch_size, max_label_length) (each row is a padded sequence of character indices). Verify that get_intseq returns padded arrays of length max_y—if it's returning 1D arrays, the labels tensor will be 2D (batch x max_length), which is correct.
• Character index range: Your final Dense layer has 28 units (for 27 characters + 1 blank token, standard for CTC). Ensure your get_intseq maps actual characters to indices 0-26, reserving index 27 for the blank token (CTC handles the blank internally—your labels should never include the blank index).
• Input length sanity: The input_length array must represent the number of MFCC time steps per audio sample (not raw audio length). Double-check that get_max_time returns the correct number of MFCC frames (e.g., for 10ms frame steps, a 1-second audio clip has 100 frames).

3. Adjust Model Structure & Training Hyperparameters #

Your model is close to DeepSpeech1, but there are tweaks to improve convergence:

• Add a second Bidirectional LSTM: DeepSpeech1 uses two stacked bidirectional LSTMs (each 2048 units). You only have one—adding a second will give the model more capacity to model sequential speech patterns:

  inner = Bidirectional(LSTM(2048, return_sequences=True))(inner)
  inner = TimeDistributed(Activation(clipped_relu))(inner)
  inner = TimeDistributed(Dropout(0.1))(inner)
  inner = Bidirectional(LSTM(2048, return_sequences=True))(inner)  # Add this line
  inner = TimeDistributed(Activation(clipped_relu))(inner)
  inner = TimeDistributed(Dropout(0.1))(inner)
  

• Lower initial learning rate: Adam's default 0.001 is likely too high for such a large model. Start with optimizer=Adam(learning_rate=1e-4) and use a learning rate scheduler to reduce it as training plateaus:

  from tensorflow.keras.callbacks import ReduceLROnPlateau
  lr_scheduler = ReduceLROnPlateau(monitor='loss', factor=0.5, patience=3, min_lr=1e-7)
  # Pass to model.fit() with callbacks=[lr_scheduler]
  

• Gradient clipping: Large models like this are prone to gradient explosion. Add clipping to your optimizer:

  optimizer=Adam(learning_rate=1e-4, clipnorm=1.0)
  

• Weight initialization: For clipped ReLU, use He-normal initialization instead of the default Glorot initialization for your Dense layers—this helps with gradient flow:

  TimeDistributed(Dense(2048, kernel_initializer='he_normal'))(inner)
  

4. Improve Training Strategy #

• Batch size: If your GPU allows, increase the batch size to 32 or 64. Larger batches provide more stable gradient estimates, which helps the model escape local minima.
• Early stopping & checkpointing: Add callbacks to halt training if loss doesn't improve and save the best-performing weights:

  from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
  early_stop = EarlyStopping(monitor='loss', patience=5, restore_best_weights=True)
  checkpoint = ModelCheckpoint('best_model.h5', monitor='loss', save_best_only=True)
  

• Validate with small data first: Test training on a tiny subset of data (e.g., 100 samples with short, repetitive transcripts). If the model can overfit this small dataset (loss drops close to 0), you know the model can learn—if not, there's still a data/label formatting issue.

5. Correct Test-Time Decoding #

Make sure you're using Keras's built-in CTC decoding function, which handles blank tokens and merges repeated characters:

# Create a test model that outputs softmax predictions
test_model = Model(inputs=input_data, outputs=output)

# Get predictions for your test data
test_x = ...  # Preprocessed MFCC data (shape: [batch_size, timesteps, 26])
y_pred = test_model.predict(test_x)

# Prepare input lengths for each test sample (number of MFCC timesteps)
input_lengths = np.array([test_x.shape[1]] * test_x.shape[0])

# Decode predictions
import tensorflow as tf
decoded = tf.keras.backend.ctc_decode(
    y_pred,
    input_length=input_lengths,
    greedy=True  # Use beam search for better results: beam_width=100
)[0][0]

# Convert decoded indices back to text (replace int_to_char with your mapping function)
decoded_text = [''.join([int_to_char(idx) for idx in seq if idx != 27]) for seq in decoded.numpy()]

Applying these fixes should help your model escape the local minimum and start learning meaningful transcriptions.

内容的提问来源于stack exchange，提问作者hhaefliger