基于Keras的嵌入式RNN：Concatenate维度问题与Cell共享实现咨询

Hey there, let's break down your two challenges and walk through solutions for each—starting with the dimension mismatch error, then moving to the shared LSTM cell for time-invariant outputs.

Your input iX is shaped (None, 30, 84)—that’s batch size, 30 one-second audio subsamples, and 84 features per subsample. The error pops up because your current CuDNNLSTM layer is only returning the final time step’s output ((None, 32)) instead of the full set of outputs for all 30 subsamples. The second RNN layer expects a 3D input (batch, timesteps, features), but you’re feeding it a 2D tensor.

The fix is straightforward: add return_sequences=True to your first CuDNNLSTM layer. This tells the layer to output all 30 time-step results instead of just the last one, giving you the (None, 30, 32) shape you need:

from tensorflow.keras.layers import CuDNNLSTM

# First LSTM: returns outputs for every 1-second subsample
first_lstm = CuDNNLSTM(32, return_sequences=True)
sequence_output = first_lstm(iX)  # Shape: (None, 30, 32)

# Now pass this to your second RNN layer (no dimension mismatch!)
second_lstm = CuDNNLSTM(64)  # Adjust size to fit your final prediction needs
final_prediction = second_lstm(sequence_output)

If you actually want to process each subsample independently (not as a sequential chain where each step depends on the previous), reshape your input to add a dummy time-step dimension for each subsample, then use TimeDistributed to apply the same LSTM to every one:

from tensorflow.keras.layers import TimeDistributed, Reshape

# Reshape input to treat each subsample as a 1-step sequence
reshaped_input = Reshape((30, 1, 84))(iX)

# Apply the same LSTM to all subsamples (weights are shared)
shared_lstm = TimeDistributed(CuDNNLSTM(32, return_sequences=False))
sequence_output = shared_lstm(reshaped_input)  # Shape: (None, 30, 32)

You want the same sound to produce the same output no matter which 1-second subsample you’re using—meaning the LSTM should use a single shared cell with a consistent state across all subsamples. This way, the output only depends on the subsample’s features, not its position in the 30-second clip.

Keras Approach #

The TimeDistributed wrapper already shares weights across all subsamples. To keep the cell state consistent (resetting it for each subsample), just use the default non-stateful LSTM (stateful=False). This ensures every subsample starts with the same initial cell state (zeros by default), and the same LSTM weights are used for all of them.

The code from the independent subsample approach above already does this—each subsample is processed with the same LSTM, starting fresh every time. If you want a custom trainable initial state instead of zeros, you can build a custom layer to define and reuse that state.

TensorFlow Low-Level Approach #

If you need full control over the cell state, use TensorFlow’s low-level API to define a single LSTM cell and loop through each subsample, resetting the state every time:

import tensorflow as tf

# Define one shared LSTM cell
lstm_cell = tf.keras.layers.LSTMCell(32)

# Split your input into 30 separate subsamples (each shape: (None, 84))
subsamples = tf.unstack(iX, axis=1)

# Initialize a consistent cell state (we'll reuse this for every subsample)
batch_size = tf.shape(iX)[0]
initial_state = lstm_cell.get_initial_state(batch_size=batch_size, dtype=tf.float32)

outputs = []
for subsample in subsamples:
    # Process the subsample with the shared cell and consistent initial state
    output, _ = lstm_cell(subsample, initial_state)
    outputs.append(output)

# Stack outputs back into a single sequence (shape: (None, 30, 32))
sequence_output = tf.stack(outputs, axis=1)

This guarantees that identical subsamples (from the same sound) will produce identical outputs, since they’re processed with the same cell and starting state.

内容的提问来源于stack exchange，提问作者Nicolas M.