请教：如何基于Gal的方法用循环Dropout实现LSTM的MC-Dropout？

Got it, let's break down how to implement MC-Dropout with recurrent Dropout for LSTMs using Keras' functional API—since you already know how to force regular Dropout to stay active during inference, this is a similar idea but tailored for the recurrent part of the LSTM.

First, a quick recap: MC-Dropout requires keeping Dropout (including recurrent Dropout) active both during training and inference to generate multiple stochastic predictions for uncertainty estimation. Keras' built-in LSTM has a recurrent_dropout parameter, but by default, it only applies when training=True (i.e., during training). So the key is to force this parameter to stay active even during inference.

1. Basic Implementation with Functional API #

Instead of adding a separate Dropout layer for the recurrent part, you can directly use the LSTM's built-in recurrent_dropout parameter and force training=True when calling the layer—just like you did with the regular Dropout layer. Here's how that looks:

import tensorflow as tf
from tensorflow.keras.layers import Input, LSTM, Dense
from tensorflow.keras.models import Model

# Define input shape (timesteps, feature count)
inputs = Input(shape=(10, 32))

# Enable recurrent Dropout and force it to stay active in all phases
# Set both `dropout` (for input-to-LSTM Dropout) and `recurrent_dropout` (for hidden state Dropout)
lstm_layer = LSTM(units=64, dropout=0.2, recurrent_dropout=0.2)
lstm_output = lstm_layer(inputs, training=True)

# Add your output layer
outputs = Dense(1, activation='sigmoid')(lstm_output)

# Build the model
model = Model(inputs=inputs, outputs=outputs)
model.compile(optimizer='adam', loss='binary_crossentropy')

2. How to Use This for MC-Dropout Inference #

Since we've set training=True on the LSTM layer, every time you run model.predict(), the recurrent Dropout will be active, producing a slightly different prediction each time. To get the MC-Dropout estimates, you just run multiple inference passes and aggregate the results:

import numpy as np

# Example test data (batch of 5 samples, 10 timesteps, 32 features)
test_data = np.random.randn(5, 10, 32)

# Generate 100 stochastic predictions
num_samples = 100
predictions = []
for _ in range(num_samples):
    pred = model.predict(test_data, verbose=0)
    predictions.append(pred)

# Calculate mean prediction (point estimate) and standard deviation (uncertainty)
mean_pred = np.mean(predictions, axis=0)
std_pred = np.std(predictions, axis=0)

print("Mean predictions:\n", mean_pred)
print("\nPrediction uncertainty (std):\n", std_pred)

3. Key Notes & Caveats #

• Don't mix up dropout and recurrent_dropout: The dropout parameter applies Dropout to the input of the LSTM layer, while recurrent_dropout applies it to the recurrent hidden state transitions. For full MC-Dropout, you'll want to enable both if needed.
• Computational cost: Recurrent Dropout adds overhead because it generates a new mask for each timestep. Running multiple inference passes for MC-Dropout will multiply this cost—plan accordingly for large datasets.
• Flexibility with training modes: If you ever need to temporarily disable Dropout (e.g., for a deterministic validation run), you can subclass the LSTM layer or use a custom model that allows toggling the training parameter dynamically. But for standard MC-Dropout, the fixed training=True approach works perfectly.

That's it! This approach mirrors what you did with the regular Dropout layer, but applies it directly to the LSTM's recurrent mechanism to enable MC-Dropout.

内容的提问来源于stack exchange，提问作者RABI Hamza