I'm doing some research for the summarization task and found out BERT is derived from the Transformer model. In every blog about BERT that I have read, they focus on explaining what is a bidirectional encoder, So, I think this is what made BERT different from the vanilla Transformer model. But as far as I know, the Transformer reads the entire sequence of words at once, therefore it is considered bidirectional too. Can someone point out what I'm missing?


2 Answers 2


The name provides a clue. BERT (Bidirectional Encoder Representations from Transformers): So basically BERT = Transformer Minus the Decoder

BERT ends with the final representation of the words after the encoder is done processing it.

In Transformer, the above is used in the decoder. That piece of architecture is not there in BERT

  • 1
    $\begingroup$ Thank you for pointing that out $\endgroup$ Nov 30, 2020 at 14:02

BERT (Bidirectional Encoder Representations from Transformers) is a deep learning model that was developed by Google in 2018. It is based on the Transformer architecture, which was introduced in the same year in the paper "Attention Is All You Need".

The main difference between BERT and the vanilla Transformer architecture is that BERT is a bidirectional model, while the Transformer is a unidirectional model. This means that BERT processes the input text in both forward and backward directions, allowing it to capture contextual information from both the left and right sides of a word. By contrast, the Transformer processes the input text in only one direction, from left to right or right to left.

Another key difference between BERT and the Transformer is that BERT is trained using a specific type of self-supervised learning called "masked language modeling". This involves masking a portion of the input text and then training the model to predict the masked tokens based on the context provided by the unmasked tokens. This allows BERT to learn general-purpose representations of language that can be fine-tuned for a wide range of tasks.

In summary, BERT is a bidirectional version of the Transformer architecture that is trained using masked language modeling, and is designed to learn general-purpose representations of language.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.