Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Nowadays many applications only use the Encoder and Decoder part of the Transformer respectively. I am having trouble understanding the difference though.
If GPT uses Decoder only and BERT uses Encoder only does this mean that the only difference between the two is basically in the masking part?
The cross attention layer in the Decoder is omitted since there is no Encoder within GPT right?
BERT is a Transformer encoder, while GPT is a Transformer decoder:
You are right in that, given that GPT is decoder-only, there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the masking in the multi-head attention block.
There is, however, an extra difference in how BERT and GPT are trained:
BERT is a Transformer encoder, which means that, for each position in the input, the output at the same position is the same token (or the [MASK] token for masked tokens), that is the inputs and output positions of each token are the same.
GPT is a Transformer decoder, which means that it is meant for autoregressive inference. This means that the tokens in the input are shifted one position to the right with respect to the output, that is, if the output is [the, dog, is, brown, </s>], the input is [<s>, the, dog, is, brown, </s>].
Apart from that, at inference time BERT generates all its output at once, while GPT is autoregressive, so you need to iteratively generate one token at a time.
