I can't see how BERT makes predictions without using a decoder unit, which was a part of all models before it including transformers and standard RNNs. How are output predictions made in the BERT architecture without using a decoder? How does it do away with decoders completely?
To put the question another way: what decoder can I use, along with BERT, to generate output text? If BERT only encodes, what library/tool can I use to decode from the embeddings?
The need for an encoder depends on what your predictions are conditioned on, e.g.:
In order to make predictions, BERT needs some tokens to be masked (i.e. replaced with a special [MASK] token. The output is generated non-autoregressively (every token at the output is computed at the same time, without any self-attention mask), conditioning on the non-masked tokens, which are present in the same input sequence as the masked tokens.
BERT is a pretraining model to do the downstream tasks such as question answering, NLI and other language tasks. So it just needs to encode the language representations so that it could be used for other tasks.That's why it consists only of encoder parts. You can add the decoder while doing your specific task and this decoder could be anything based on your task.
In short, Bidirectional Encoder Representations from Transformers (BERT) is not designed for decorder-related tasks.
I can't see how BERT makes predictions without using a decoder unit, which was a part of all models before it including transformers and standard RNNs. How are output predictions made in the BERT architecture without using a decoder? How does it do away with decoders completely?
If I undertand correctly, you are asking how BERT predicts its output. If you read the paper, BERT uses a technique called masked language modeling (MLM). During training, BERT randomly masks some of the tokens in a sentence and then tries to predict what the original word was. For example, in the sentence "I want to buy a [MASK]", BERT might be trained to predict that the masked word is "car" based on the context of the other words in the sentence.
The MLM task only requires an encoder because it involves encoding the input text into a fixed-length vector representation that can be used for downstream tasks such as sentiment analysis, question-answering, and language generation.
BERT also uses another technique called next sentence prediction. During training, BERT is given pairs of sentences and is asked to predict whether the second sentence follows logically from the first. This allows BERT to capture the relationships between sentences and to understand the broader context of a given text.
During inference, BERT uses its pre-trained weights to encode the input sentence and then passes the encoded representation through one or more neural network layers to generate the output. The exact details of how BERT generates its output depend on the specific task it is being used for, but in general, it uses a combination of the encoded input, attention mechanisms, and learned weights to produce the final output.
To put the question another way: what decoder can I use, along with BERT, to generate output text? If BERT only encodes, what library/tool can I use to decode from the embeddings?
I see you have edited your questions. In this newly added question, I can see that you have found that BERT only provides you with embeddings. You need other architectures like decorders along with these embeddings for your downstream tasks.
Here are some examples from the smiley face https://huggingface.co/learn/nlp-course/chapter1/6
First need to understand what problems BERT can solve or what kind of inference/prediction it can achieve.
Encoder in Transformer itself can learn:
Relations among words (what word is most probable in a context). For instance, what word will fit in the BLANK in the context I take [BLANK] of the opportunity.
Relations among sentences. For instance A: "In a glossary store" can follow B: "I bought the ingredients".
Having these traits or capabilities, BERT can predict a word to follow a sequence of words. BERT can classify if a text is negative or positive. As far as you can achieve the predictions you want with the Encoder part only, you do not need the Decoder.
Hence it would be better focus on what problems require Decoder. Or what problems BERT cannot solve.
I only understood tranformer decoders and encoders after I watched this video. Check it out:
BERT (Bidirectional Encoder Representations from Transformers) is a transformer-based language model that is trained to predict the next word in a sequence given the context of the previous words. It does this by using a bidirectional encoder to process the input text and generate a fixed-length representation of the input. The decoder is not a part of the BERT architecture because it is not designed to generate text as output. Instead, it is used to encode the input text into a fixed-length representation that can be fed into a downstream task such as question answering or language translation.
In a typical language model, the decoder is responsible for generating text as output based on the context provided by the encoder. However, in the case of BERT, the encoder is trained to generate a fixed-length representation of the input text, and this representation is used as input to a downstream task that is responsible for generating text as output. The BERT model itself does not have a decoder component and is not designed to generate text directly.
BERT is a stack of deep bidirectional transformer encoders that read the input sequence and generate meaning representations called embeddings. It uses multi-head attention to decide the meaning.
