I don't understand the difference in mechanics of a transformer vs LSTM for a sequence prediction problem. Here is what I have gathered so far:



suppose we want to predict the remaining tokens in the word 'deep' given the first token 'd'. Then the first input will be 'd', and the predicted output is 'e'. Now at the next time step, the previously predicted output 'e' is fed along with the previous hidden state which contains information on 'd'. This is done till the predicted output is


In the same example, how would the transformer work, and avoid sequential inputs? Would we be giving the entire word 'deep' as input and leave it to the network to get the characters in correct sequence, or do we only input 'd' (which is what we did in LSTM)?

I'm really confused and I think I am missing out on some very fundamental concepts here. Would be really thankful for your help. Thanks!


2 Answers 2


First of all, I would not consider each letter as a token of your input sequence, think of the words as a whole as your tokens.

Regarding the problem of predicting the next token (word) given some input sequence, the accepted architecture nowadays is sequence-to-sequence with encoder-decoder, where you encode your input sequence (source sentence) into one or several vectors, which is then fed into the decoder (together with the former outputs).

If you try to predict the next token with a usual step-by-step LSTM based only on former input tokens, without any context of the whole sentence, it might be not possible to predict something reasonable when having not enough words yet (think of a translation machine trying to predict a 2nd or 3rd word based only on the first or 2 first words), where each output token N is based on the input tokens 0...N + the N-1 output tokens predicted by that step: enter image description here

but in a proper sequence-to-sequence approach, you better encode your whole input sequence into a single representation, which is fed into the decoder (which can have a GRU, lSTM...):

enter image description here

but there is still one problem with this sencond approach, which is dealing with the whole input sequence and, for very long input sentences, it might be difficult for a RNN to retain all the neccessary info, loosing then some context. Here is where attention based transformer models comes in to play:

enter image description here

where each token is encoded via attention mechanism, giving words representations a context meaning. The decoder of the transformer model uses neural attention to identify tokens of the encoded source sentence which are closely related to the target token to predict.

A great source of info about this all is the second edition of the book Deep learning with python by François Chollet.

  • $\begingroup$ I highly disagree with your statement: If you try to predict the next token with a usual step-by-step LSTM based only on former input tokens, without any context of the whole sentence, it might be not possible to predict something reasonable, where each output token N is based on the input tokens 0...N + the N-1 output tokens predicted by that step. It is perfectly possible to generate reasonable text, with a decoder-only transformer, or a single LSTM, like what the OP asked. This is what language models do, including how GPT-3 (Transformer-based) and older LSTM-based LMs generate text. $\endgroup$
    – noe
    Sep 5, 2021 at 14:11
  • $\begingroup$ And I disagree with your understanding of what I said. I did not mean you cannot predict correctly the next token, I meant you MIGHT not be able to, because when you are predicting the first tokens with a RNN, you might not have yet enough context $\endgroup$
    – German C M
    Sep 5, 2021 at 15:16
  • $\begingroup$ I see your point. $\endgroup$
    – noe
    Sep 5, 2021 at 16:04
  • $\begingroup$ Thanks for your feedback $\endgroup$
    – German C M
    Sep 5, 2021 at 16:07
  • $\begingroup$ So the input is actually a certain number of tokens (lets say 3 words), then using these 3 words the model predicts the 4th, and then combines all this info to predict the 5th word and so on...is that right? $\endgroup$
    – huy
    Sep 6, 2021 at 5:55

Either at training time or at inference time, both an LSTM and a Transformer decoder act exactly the same in terms of inputs and outputs:

At training time, you provide the whole sequence as input, and you obtain the next token predictions. In LSTMs, this training regime is called "teacher forcing"; we use this fancy name because LSTMs (RNNs in general) have another way of being trained: by using at training time their own predictions as input for the following token. This training regime, however, is unstable and almost nobody uses it in practice; the standard for training LSTMs is using teacher forcing.

At inference time, you predict one token at a time, and use such a prediction as the input for the next step.

Note that I am referring to the Transformer decoder, not the whole encoder-decoder Transformer, which was originally proposed by Vaswani et al., 2017:

In an encoder-decoder Transformer (e.g. source sentence in a neural machine translation setup), the encoder part ingests the context at once, processing it in parallel, and the decoder receives the output of the encoder and acts autoregressively, as I described previously.

You can't compare a single LSTM with an encoder-decoder Transformer architecture. But you can indeed compare the text generation process of an LSTM an a Transformer decoder. Both are normally used as language models.

  • $\begingroup$ Although there are quite a lot similarities, in my humble opinion, saying LSTMs and Transformers act the same in terms of inputs is misleading for someone who did not see it before, since in transformers, you feed the input encoded as a whole with attention as the key advantage, and not as a step by step frame $\endgroup$
    – German C M
    Sep 5, 2021 at 15:19
  • 1
    $\begingroup$ I added the clarification "Transformer" --> "Transformer decoder", which is what I meant. I also added a clarification about comparing an LSTM to an encoder-decoder Transformer. Thanks! $\endgroup$
    – noe
    Sep 5, 2021 at 15:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.