0
$\begingroup$

If BERT is a stack of transformer encoders, and the encoder already operates bidirectionally, understanding both left and right contexts and generating contextual embeddings, what is the purpose of pretraining BERT using MLM ? Does it aim to improve the contextual embeddings even better ? Could someone please provide an explanation on this ? Thanks.

$\endgroup$

1 Answer 1

0
$\begingroup$

When you train any model, you need to train it on a specific task and its associated loss function. Masked-language modelling is the token-level task used for training transformer encoders like BERT and RoBERTa on unlabeled text. While the transformer encoder architecture enables it to "operate bidirectionally", you still need a loss that exercises such bidirectionality so that the model is actually trained.

Therefore, you can't have a BERT model (nor BERT contextual embeddings) unless you train it in some task like masked language modelling.

$\endgroup$
10
  • $\begingroup$ Why didn't they just use transformer encoder right away, since it's already work bidirectionally, I mean why did they have to train ? $\endgroup$
    – user159173
    Mar 12 at 10:53
  • $\begingroup$ And about MLM, I read somewhere saying MLM is for producing contextual embeddings because the model process left and right context of the mask, so BERT contextual embedding is better than vanila transformer encoder's contextual embedding ? $\endgroup$
    – user159173
    Mar 12 at 10:54
  • $\begingroup$ You need to train the model using a loss, and only then can you extract embeddings from the trained model. When you say "Vanilla transformer encoder's contextual embeddings", you are not specifying a loss, so it does not make sense to say so. $\endgroup$
    – noe
    Mar 12 at 11:06
  • $\begingroup$ Are you saying that if I only use the vanilla transformer's encoder, I can't extract the contextual embedding ? $\endgroup$
    – user159173
    Mar 12 at 11:09
  • $\begingroup$ No. I am saying that BERT is a vanilla transformer encoder trained on a MLM task and a NSP task, and RoBERTa is a vanilla transformer encoder trained on a MLM task, and that just saying "vanilla transformer encoder" says nothing about the model and it can refer to any transformer encoder. $\endgroup$
    – noe
    Mar 12 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.