Please explain Transformer vs LSTM using a sequence prediction example

Question

I don't understand the difference in mechanics of a transformer vs LSTM for a sequence prediction problem. Here is what I have gathered so far:

LSTM:

suppose we want to predict the remaining tokens in the word 'deep' given the first token 'd'. Then the first input will be 'd', and the predicted output is 'e'. Now at the next time step, the previously predicted output 'e' is fed along with the previous hidden state which contains information on 'd'. This is done till the predicted output is

Transformer:

In the same example, how would the transformer work, and avoid sequential inputs? Would we be giving the entire word 'deep' as input and leave it to the network to get the characters in correct sequence, or do we only input 'd' (which is what we did in LSTM)?

I'm really confused and I think I am missing out on some very fundamental concepts here. Would be really thankful for your help. Thanks!

Answer 1

First of all, I would not consider each letter as a token of your input sequence, think of the words as a whole as your tokens.

Regarding the problem of predicting the next token (word) given some input sequence, the accepted architecture nowadays is sequence-to-sequence with encoder-decoder, where you encode your input sequence (source sentence) into one or several vectors, which is then fed into the decoder (together with the former outputs).

If you try to predict the next token with a usual step-by-step LSTM based only on former input tokens, without any context of the whole sentence, it might be not possible to predict something reasonable when having not enough words yet (think of a translation machine trying to predict a 2nd or 3rd word based only on the first or 2 first words), where each output token N is based on the input tokens 0...N + the N-1 output tokens predicted by that step:

but in a proper sequence-to-sequence approach, you better encode your whole input sequence into a single representation, which is fed into the decoder (which can have a GRU, lSTM...):

but there is still one problem with this sencond approach, which is dealing with the whole input sequence and, for very long input sentences, it might be difficult for a RNN to retain all the neccessary info, loosing then some context. Here is where attention based transformer models comes in to play:

where each token is encoded via attention mechanism, giving words representations a context meaning. The decoder of the transformer model uses neural attention to identify tokens of the encoded source sentence which are closely related to the target token to predict.

A great source of info about this all is the second edition of the book Deep learning with python by François Chollet.

Answer 2

Either at training time or at inference time, both an LSTM and a Transformer decoder act exactly the same in terms of inputs and outputs:

At training time, you provide the whole sequence as input, and you obtain the next token predictions. In LSTMs, this training regime is called "teacher forcing"; we use this fancy name because LSTMs (RNNs in general) have another way of being trained: by using at training time their own predictions as input for the following token. This training regime, however, is unstable and almost nobody uses it in practice; the standard for training LSTMs is using teacher forcing.

At inference time, you predict one token at a time, and use such a prediction as the input for the next step.

Note that I am referring to the Transformer decoder, not the whole encoder-decoder Transformer, which was originally proposed by Vaswani et al., 2017:

In an encoder-decoder Transformer (e.g. source sentence in a neural machine translation setup), the encoder part ingests the context at once, processing it in parallel, and the decoder receives the output of the encoder and acts autoregressively, as I described previously.

You can't compare a single LSTM with an encoder-decoder Transformer architecture. But you can indeed compare the text generation process of an LSTM an a Transformer decoder. Both are normally used as language models.