How is the LSTM RNN forget gate calculated?

Question

I understand that the LSTM recurrent neural network has a forget gate, input gate, and output gate.

However, I do not understand how the equation below calculates 'forget' information (from Chris Olah's LSTM post)

How does this forget gate decide which information is discarded from the previous cell state? In other words, what is forgotten or not.

It appears to me that this equation just takes $x_t$ from the current state and $h_{t-1}$ from the previous state.

Can someone explain how this equation works as a forget gate to retain or discard cell state?

Answer 1

The equation and value of $f_t$ by itself does not fully explain the gate. You need to look at first term of the next step:

$C_t = f_t \odot C_{t-1} + i_t \odot \bar{C}_{t}$

The vector $f_t$ that is the output from the forget gate, is used as element-wise multiply against the previous cell state $C_{t-1}$. It is this stage where individual elements of $C$ are "remembered" or "forgotten".

Due to the sigmoid function, the vector $f_t$ behaves like a binary classifier for each element, with saturated values tending to settle on either not modifying $C$ at all (a value of $1$) or "forgetting" what the previous value was (a value of $0$). Of course intermediate values are also possible, and the analogy there between simply remembering and forgetting values is less direct.

The analogy with forgetting or remembering helps towards understanding the improvements regarding preserving gradients over multiple timesteps. For any step where an element of $f_t$ is close to $1$ (and thus the effect of previous timesteps is being "remembered"), then the corresponding gradient elements backpropagating from $\nabla_{\theta} C_{t}$ to $\nabla_{\theta} C_{t-1}$ remain the same, avoiding the gradient loss seen in more simple RNN architectures (especially related to saturated values). Once a previous element in $C$ is "forgotten", then the error gradient connection is cut off between time steps.