0
$\begingroup$

I can't see how BERT makes predictions without using a decoder unit, which was a part of all models before it including transformers and standard RNNs. How are output predictions made in the BERT architecture without using a decoder? How does it do away with decoders completely?

$\endgroup$
3
$\begingroup$

The need for an encoder depends on what your predictions are conditioned on, e.g.:

  • In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given that the previous tokens are received by the decoder itself, you don't need an encoder.
  • In Neural Machine Translation (NMT) models, each token of the translation is predicted conditioning on the previous tokens and the source sentence. The previous tokens are received by the decoder, but the source sentence is processed by a dedicated encoder. Note that this is not necessarily this way, as there are some decoder-only NMT architectures, like this one.
  • In masked LMs, like BERT, each masked token prediction is conditioned on the rest of the tokens in the sentence. These are received in the decoder, therefore you don't need an encoder. This, again, is not a strict requirement, as there are other masked LM architectures, like MASS that are encoder-decoder.

In order to make predictions, BERT needs some tokens to be masked (i.e. replaced with a special [MASK] token. The output is generated non-autoregressively (every token at the output is computed at the same time, without any self-attention mask), conditioning on the non-masked tokens, which are present in the same input sequence as the masked tokens.

|improve this answer|||||
$\endgroup$
-1
$\begingroup$

BERT uses a decoder unit.. check at time 12 minutes in this video - BERT Transformer architecture explanation

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.