I am reading this paper "Sequence to Sequence Learning with Neural Networks" http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf

Under "2. The Model" it says:

The LSTM computes this conditional probability by first obtaining the fixed dimensional representation v of the input sequence (x1, . . . , xT ) given by the last hidden state of the LSTM, and then computing the probability of y1, . . . ,yT′ with a standard LSTM-LM formulation whose initial hidden state is set to the representation v of x1, . . . , xT:

I know what an LSTM is, but what's an LSTM-LM? I've tried Googling it but can't find any good leads.

  • $\begingroup$ But this sentence is still puzzling to me. if I put it into equation if makes ![](i.sstatic.net/0Lv8L.png) ![(https://i.sstatic.net/et5Sf.png) with c the last hidden state of the encoder. then the first hidden state represents the information provided by the encoder but the next ones represent the probability distribution of the target sequence's elements : something of a radically different nature. Also the cell state state initialisation is not given and the figure 1 let believe that the LSTM provid $\endgroup$ Sep 18, 2018 at 12:40

2 Answers 2


The definition of a Language Model (LM) is a probability distribution over sequences of words.

The simple illustration of an LM is predicting the next word given the previous word(s).

For example, if I have a language model and some initial word(s):

  • I set my initial word to My
  • My model predicts there is a high probability that name appears after My.
  • By setting the initial words to My name, my model predicts there is a high probability that is appears after My name.
  • So it's like: My -> My name -> My name is -> My name is Tom, and so on.

You can think of the autocompletion on your smartphone keyboard. In fact, LM is the heart of autocompletions.

So, LSTM-LM is simply using an LSTM (and softmax function) to predict the next word given your previous words.

By the way, Language Model is not limited to LSTM, other RNNs (GRU), or other structured models. In fact, you can also use feedforward networks with context/sliding/rolling window to predict the next word given your initial words.

  • $\begingroup$ Does that change the formulation of the LSTM itself in any way? $\endgroup$ Aug 4, 2016 at 8:37
  • $\begingroup$ Or does it change the way that the LSTMs are linked together? $\endgroup$ Aug 4, 2016 at 8:37
  • 1
    $\begingroup$ IMHO, perhaps it means an LSTM that is tuned for LM (Language Modeling). I am reading the same paper and that is my understanding $\endgroup$
    – Ali
    Oct 24, 2016 at 15:33
  • $\begingroup$ @TaevanbatMongol no it's not changing the LSTM formulation. You only need a softmax function (or something) to generate the probability of words from the LSTM output $\endgroup$ Oct 15, 2017 at 14:38
  • $\begingroup$ Probability of words means if you sum the probability/score of the output of a timestep, it will equals to 1 $\endgroup$ Oct 15, 2017 at 14:39

In this context I think it means you take the output representation and learn an additional softmax layer that corresponds to the tokens in your language model (in this case letters).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.