0
$\begingroup$

How do BERT and RoBERTa generate contextual embeddings? The articles I've read keep saying that transformer encoders work bidirectionally. Because of self-attention, they can look at every token, unlike RNN/LSTM, which can only process the previous hidden state. Is it true ? I'm not sure how BERT and RoBERTa accomplish that.

$\endgroup$
3
  • 2
    $\begingroup$ Does this answer your question? How Transformer is Bidirectional - Machine Learning $\endgroup$
    – noe
    Mar 20 at 19:01
  • $\begingroup$ I'm actually looking for the mathematical explanation instead of just reading claims that it's bidirectional, attends to every token, and produces contextual embeddings, etc. It would be helpful if I could understand the entire process from input to output of BERT or RoBERTa that creates contextual embeddings $\endgroup$
    – abcd
    Mar 20 at 19:12
  • 1
    $\begingroup$ You can find the mathematical explanations in the original Transformers article together with the BERT article. For a friendlier version you can check the Illustrated BERT blog post. $\endgroup$
    – noe
    Mar 20 at 19:29

1 Answer 1

0
$\begingroup$

Start with a sequence of embeddings.

In the standard attention computation, each embedding in the sequence attends to every other embedding in the sequence. This can be considered "bidirectional" as a given embedding is attending every other embedding, considering the full sequence.

Masked language models (MLMs) like BERT use this form of attention.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.