I know in a LSTM chain you should connect the h(t) of the previous cell to the h(t+1) of the next cell, and doing so for c(t). But what about the first cell? What does it get as h(t-1) and c(t-1)?

I also like to know, if we want to make a multi layer LSTM, What should we give to the first cell of the second layer as h and c? Also will we throw away the h and c of the last cell of the each layer?

And a little other question: will we give the y(output) of each cell to the x(input) of it's corresponding cell in the top layer?


Assuming that the LSTM is going to be used for sequence generation (e.g. in a language model or the decoder part of an encoder-decoder NMT architecture), we have the following:

In supervised learning setups:

In a language model, the LSTM's $h_{-1}$ and $c_{-1}$ are initialized to zeros, for all the layers. If the LSTM is the decoder part of an encoder-decoder NMT architecture, they will be initialized with the hidden state and context vector of the LSTM from the encoder part, either from the last position of from a combination of all positions by using some attention mechanism.

During training, the inputs $x$ are not fed with the LSTM predictions for the previous tokens because this presents convergence problems; instead, you use teacher forcing, which consists in feeding the gold tokens (the tokens from the training data) as inputs.

During inference, you have to use the network's own predictions as inputs for the next tokens.

The mismatch between the training and inference regimes is said to present exposure bias, that is, during training you make the network see only "good" previous tokens and it never learns to recover from its own bad predictions. However, according to recent research, exposure bias may actually not be a serious problem.

In more exotic learning setups:

In textual GANs, $h_{-1}$ and $c_{-1}$ are sometimes initialized with a latent vector $z$ that follows some prior distribution.

Also, in textual GANs and other learning setups where there are no gold tokens, teacher forcing is not an option, so you need to use the LSTM predictions as previous token inputs.

  • $\begingroup$ beautiful answer, +1 $\endgroup$ – pcko1 Jul 5 '19 at 7:45
  • $\begingroup$ Well, nice explanation, almost. I'd like to point out that the teacher forcing technique is used in the decoder of a sequence to sequence model, since in the encoder the model has indeed access to the golden tokens. It may be obvious for experienced users, but I think it is worth of comment. For further details this is a nice introduction to sequence-sequence models in keras: blog.keras.io/… $\endgroup$ – ignatius Jul 5 '19 at 8:25
  • $\begingroup$ Thanks for pointing that out, I'll clarify in the answer $\endgroup$ – noe Jul 5 '19 at 8:26
  • $\begingroup$ You're very welcome :) $\endgroup$ – ignatius Jul 5 '19 at 8:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.