2
$\begingroup$

Nowadays many applications only use the Encoder and Decoder part of the Transformer respectively. I am having trouble understanding the difference though.

If GPT uses Decoder only and BERT uses Encoder only does this mean that the only difference between the two is basically in the masking part?

The cross attention layer in the Decoder is omitted since there is no Encoder within GPT right?

$\endgroup$

1 Answer 1

5
$\begingroup$

BERT is a Transformer encoder, while GPT is a Transformer decoder:

transformer_diagram

You are right in that, given that GPT is decoder-only, there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the masking in the multi-head attention block.

There is, however, an extra difference in how BERT and GPT are trained:

  • BERT is a Transformer encoder, which means that, for each position in the input, the output at the same position is the same token (or the [MASK] token for masked tokens), that is the inputs and output positions of each token are the same.

  • GPT is a Transformer decoder, which means that it is meant for autoregressive inference. This means that the tokens in the input are shifted one position to the right with respect to the output, that is, if the output is [the, dog, is, brown, </s>], the input is [<s>, the, dog, is, brown, </s>].

Apart from that, at inference time BERT generates all its output at once, while GPT is autoregressive, so you need to iteratively generate one token at a time.

$\endgroup$
3
  • $\begingroup$ thank you. That was very concise and clear. thinking of Vision Transformers, if I have two different streams of images (i.e. multimodal) and I want to relate them to each other (in feature space) would I use to encoders (one for each modality) and then compare them somehow or would I use one encoder for the first image stream and the second stream as input to the Decoder that then attends over the encodes 1st stream? $\endgroup$
    – CD86
    Jan 8, 2021 at 14:04
  • 1
    $\begingroup$ Sorry, I know very little about image transformers. I suggest you create a new question for that, so that people familiar with the topic can help. $\endgroup$
    – noe
    Jan 8, 2021 at 15:04
  • 1
    $\begingroup$ Also, if you found my answer helpful, please consider upvoting and marking it as correct. $\endgroup$
    – noe
    Jan 8, 2021 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.