What is the difference between GPT blocks and Transformer Decoder blocks?

Question

I know GPT is a Transformer-based Neural Network, composed of several blocks. These blocks are based on the original Transformer's Decoder blocks, but are they exactly the same?

In the original Transformer model, Decoder blocks have two attention mechanisms: the first is pure Multi Head Self-Attention, the second is Self-Attention with respect to Encoder's output. In GPT there is no Encoder, therefore I assume its blocks only have one attention mechanism. That's the main difference I found.

At the same time, since GPT is used to generate language, its blocks must be masked, so that Self-Attention can only attend previous tokens. (Just like in Transformer Decoders.)

Is that it? Is there anything else to add to the difference between GPT (1,2,3,...) and the original Transformer?

Answer 1

GPT uses an unmodified Transformer decoder, except that it lacks the encoder attention part. We can see this visually in the diagrams of the Transformer model and the GPT model:

For GPT-2, this is clarified by the authors in the paper:

There have been several lines of research studying the effects of having the layer normalization before or after the attention. For instance the "sandwich transformer" tries to study different combinations.

For GPT-3, there are further modifications on top of GPT-2, also explained in the paper:

Answer 2

In original Transformer the decoder cannot attend to previous token while encoder can attend to all tokens.

Also, output of encoder is same number of tokens as input while output of decoder is just one token.

For GPT/BERT, this is the main differentiator that leads to calling them decoder-only vs encoder-only models. There is no cross attention involved in either of them.

BERT generates same number of tokens as input that can be fed to linear layer and uses masked language modeling so this is strictly encoder only model.

GPT generates one token at a time just like decoder of transformer and has causal language modeling so it is strictly decoder only model.

For completeness, there are indeed architectures with only decoder but using masked language modeling but they show less of zero shot perf. There are also encoder-decoder architectures like T5 which again doesn’t do as good as decoder-only architectures like GPT3 unless you train on supervised datasets as well (example T0 generated from T5).

Reference: https://arxiv.org/abs/2204.05832