I understand that the LSTM recurrent neural network has a forget gate, input gate, and output gate.

However, I do not understand how the equation below calculates 'forget' information (from Chris Olah's LSTM post)

LSTM forget gate layer and equation

How does this forget gate decide which information is discarded from the previous cell state? In other words, what is forgotten or not.

It appears to me that this equation just takes $x_t$ from the current state and $h_{t-1}$ from the previous state.

Can someone explain how this equation works as a forget gate to retain or discard cell state?

  • $\begingroup$ Neil Slater's answer is true. These gates are intuitively meant so. For example a classic case is, remembering "Gender" and passing it in states. E.g. She came out and played with her dog. In the above example, "she" is useful in determining "her". But how do we know which words to concentrate and are useful in remembering gender. Thats what the LSTM network will learn. Especially in text, when we use word embeddings, it is pretty straight to learn such relationships using Forget gate $\endgroup$ – Sathyanarayanan Kulasekaran Jan 10 '19 at 5:24

The equation and value of $f_t$ by itself does not fully explain the gate. You need to look at first term of the next step:

$C_t = f_t \odot C_{t-1} + i_t \odot \bar{C}_{t}$

The vector $f_t$ that is the output from the forget gate, is used as element-wise multiply against the previous cell state $C_{t-1}$. It is this stage where individual elements of $C$ are "remembered" or "forgotten".

Due to the sigmoid function, the vector $f_t$ behaves like a binary classifier for each element, with saturated values tending to settle on either not modifying $C$ at all (a value of $1$) or "forgetting" what the previous value was (a value of $0$). Of course intermediate values are also possible, and the analogy there between simply remembering and forgetting values is less direct.

The analogy with forgetting or remembering helps towards understanding the improvements regarding preserving gradients over multiple timesteps. For any step where an element of $f_t$ is close to $1$ (and thus the effect of previous timesteps is being "remembered"), then the corresponding gradient elements backpropagating from $\nabla_{\theta} C_{t}$ to $\nabla_{\theta} C_{t-1}$ remain the same, avoiding the gradient loss seen in more simple RNN architectures (especially related to saturated values). Once a previous element in $C$ is "forgotten", then the error gradient connection is cut off between time steps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.