如何为语音情感识别双向LSTM模型添加Attention层及参数疑问

Hey Leo, let's tackle your questions about adding an Attention layer to your Bi-LSTM speech emotion recognition model one by one!

1. Integrating an Attention Layer into Your Bi-LSTM Model #

Your current sequential model works well, but adding an Attention layer requires a bit of flexibility—the Functional API is better suited here because the Attention layer needs explicit input pairs (query and value). Here are two practical approaches:

Option 1: Use TensorFlow's Built-in Attention Layer #

This is the simplest way if you don't need custom logic. We'll adjust your model to use the Functional API, insert the Attention layer after the first Bidirectional LSTM (since it returns sequences), then proceed with your existing layers:

import tensorflow as tf
from tensorflow.keras.layers import Input, Masking, Bidirectional, LSTM, Dense, Attention
from tensorflow.keras.models import Model

def bi_lstm_with_attention_model(X_train, y_train, X_test, y_test, num_classes,
                                 batch_size=68, units=128, learning_rate=0.005, 
                                 epochs=20, dropout=0.2, recurrent_dropout=0.2,
                                 attention_dropout=0.1):
    class myCallback(tf.keras.callbacks.Callback):
        def on_epoch_end(self, epoch, logs={}):
            # Note: Updated 'acc' to 'accuracy' to match modern Keras metrics naming
            if logs.get('accuracy') > 0.95:
                print("\nReached 95% accuracy—stopping training early!")
                self.model.stop_training = True
    callbacks = myCallback()

    # Build model with Functional API
    inputs = Input(shape=(X_train.shape[1], X_train.shape[2]))
    x = Masking(mask_value=0.0)(inputs)
    
    # First Bidirectional LSTM (returns sequences for Attention)
    x = Bidirectional(LSTM(units, dropout=dropout, recurrent_dropout=recurrent_dropout, return_sequences=True))(x)
    
    # Add Attention layer: use the sequence as both query and value (self-attention)
    attention_layer = Attention(dropout=attention_dropout, use_scale=True)  # Enable learnable scale if needed
    attended_output = attention_layer([x, x])
    
    # Second Bidirectional LSTM (processes attended sequence)
    x = Bidirectional(LSTM(units, dropout=dropout, recurrent_dropout=recurrent_dropout))(attended_output)
    
    # Final classification layer
    outputs = Dense(num_classes, activation='softmax')(x)
    
    model = Model(inputs=inputs, outputs=outputs)
    
    # Compile and train
    adamopt = tf.keras.optimizers.Adam(learning_rate=learning_rate, beta_1=0.9, beta_2=0.999, epsilon=1e-8)
    model.compile(loss='binary_crossentropy', optimizer=adamopt, metrics=['accuracy'])
    history = model.fit(X_train, y_train, batch_size=batch_size, epochs=epochs, 
                        validation_data=(X_test, y_test), verbose=1, callbacks=[callbacks])
    score, acc = model.evaluate(X_test, y_test, batch_size=batch_size)
    yhat = model.predict(X_test)
    return history, yhat

Option 2: Custom Attention Layer (for Full Control) #

If you want to visualize attention weights or implement custom logic (like additive attention), you can define your own layer:

class CustomSelfAttention(tf.keras.layers.Layer):
    def __init__(self, units):
        super().__init__()
        self.W1 = Dense(units)
        self.W2 = Dense(units)
        self.V = Dense(1)

    def call(self, inputs):
        # inputs shape: (batch_size, timesteps, features)
        query = self.W1(inputs)  # (batch_size, timesteps, units)
        key = self.W2(inputs)    # (batch_size, timesteps, units)
        
        # Compute attention scores
        score = self.V(tf.nn.tanh(query + key))  # (batch_size, timesteps, 1)
        attention_weights = tf.nn.softmax(score, axis=1)  # Normalize over timesteps
        
        # Compute context vector (weighted sum of sequence features)
        context_vector = attention_weights * inputs
        context_vector = tf.reduce_sum(context_vector, axis=1)  # (batch_size, features)
        
        return context_vector, attention_weights

To use this in your model, replace the built-in Attention layer with:

context_vector, attention_weights = CustomSelfAttention(units)(x)
# Optional: Add a dense layer after the context vector
x = Dense(units, activation='relu')(context_vector)

2. Valid Parameters for the Attention Layer #

Yes, use_scale, causal, and dropout are all valid parameters for TensorFlow's tf.keras.layers.Attention:

• dropout: A float between 0 and 1. Applies dropout to the attention weights, preventing the model from over-relying on specific timesteps.
• use_scale: Boolean (default: False). If True, multiplies attention scores by a learnable scalar to help the model adjust attention intensity, which can stabilize training.
• causal: Boolean (default: False). Enforces causal attention, where each timestep can only attend to previous timesteps. This is useful for sequence generation tasks (e.g., text) but not necessary for speech emotion recognition—you want the model to use the entire sequence for context.

3. Coordinating Attention Dropout with LSTM Dropout #

These dropout mechanisms target different parts of the model, so you can tune them independently with these guidelines:

• Roles differ:
  • LSTM dropout: Applies dropout to the input features of the LSTM cells.
  • LSTM recurrent_dropout: Applies dropout to the recurrent state (previous timestep's output). Note: This may conflict with CuDNN-accelerated LSTMs—set to 0 if using CuDNN.
  • Attention dropout: Applies dropout directly to the attention weights.
• Start with conservative values: If your LSTM uses dropout=0.2, set Attention dropout to 0.1–0.15 to avoid over-regularization.
• Tune based on validation performance:
  • If validation accuracy is volatile (overfitting), increase Attention dropout slightly.
  • If both training and validation accuracy are low (underfitting), reduce all dropout values.
• Avoid extreme values: Don't set all dropout rates above 0.3—this will make it hard for the model to learn meaningful patterns.

Quick note: Double-check your loss function—if num_classes > 2, use categorical_crossentropy instead of binary_crossentropy for better performance.

内容的提问来源于stack exchange，提问作者Leo