LSTM单元反向计算技术问询：输入输出互换后的特性验证

Great question—let’s break this down into two key parts to unpack your thought experiment.

1. Does a single LSTM unit retain nonlinearity when reversing input/output? #

Short answer: Yes, the relationship between $h'_t$ and $x'_t$ remains nonlinear.

Let’s ground this in standard LSTM forward equations (we’ll assume fixed pre-computed $C_{t-1}$ and $h_{t-1}$ since the unit has already completed forward propagation):

• Output gate: $o_t = \text{sigmoid}(W_{ox}x_t + b_o + W_{oh}h_{t-1} + b_{oh})$
• Cell state: $C_t = f_tC_{t-1} + i_t\text{tanh}(W_{xx}x_t + b_x + W_{xh}h_{t-1} + b_h)$ where $f_t = \text{sigmoid}(W_{fx}x_t + b_f + W_{fh}h_{t-1} + b_f)$ and $i_t = \text{sigmoid}(W_{ix}x_t + b_i + W_{ih}h_{t-1} + b_i)$
• Hidden state: $h_t = o_t * \text{tanh}(C_t)$

When reversing this flow (feeding $h'_t$ to derive $x'_t$), you’re working backwards through stacked nonlinear operations:

1. First, $h'_t = o_t * \text{tanh}(C_t)$—both $o_t$ (sigmoid) and $\text{tanh}(C_t)$ are nonlinear functions of $x_t$.
2. Solving for $x'_t$ requires unrolling the sigmoid and tanh layers that define $o_t$, $i_t$, $f_t$, and $C_t$. Even though sigmoid and tanh have inverse functions, the composite mapping from $h'_t$ to $x'_t$ stays nonlinear because it’s a chain of non-proportional transformations. Changing $h'_t$ will lead to a non-linear shift in $x'_t$ thanks to the saturation effects of sigmoid/tanh and the multiplicative interactions between gates and cell state.

2. Is reversing only the output gate sufficient for the entire sequence? #

Short answer: No, you can’t rely solely on the output gate to compute $x'_t$ across the sequence.

The hidden state $h_t$ depends on two critical components: the output gate $o_t$ and the cell state $C_t$. While $o_t$ directly modulates $h_t$, $C_t$ is determined by the interplay of the input gate (controls new information entering the cell) and 遗忘门 (controls old information retained from $C_{t-1}$). Both gates are directly tied to $x_t$ via their weight matrices $W_{ix}$ and $W_{fx}$.

If you only reverse the output gate, you’d only capture the portion of $x_t$ that affects $o_t$, but you’d ignore the larger contribution of $x_t$ to $C_t$ through $i_t$ and $f_t$. For example:

• Even if you solve $o_t = h'_t / \text{tanh}(C_t)$, you still can’t determine $C_t$ without accounting for $i_t$ and $f_t$, which require $x_t$ to compute.
• Across a sequence, each $C_t$ feeds into $C_{t+1}$, so ignoring input/遗忘 gates breaks the chain of dependencies needed to back-derive consistent $x'_t$ values for every unit.

To get valid $x'_t$ values for the entire sequence, you need to reverse-engineer all three gates (input,遗忘, output) along with cell state updates—you can’t skip the first two.

内容的提问来源于stack exchange，提问作者wiggalicious