4
$\begingroup$

I am reading the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding that can be found here.

It looks to me that the crux of the paper is using masked inputs to achieve bidirectionally.

This is an excerpt from the Google AI blog here which states:

"However, it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that’s being predicted to indirectly “see itself” in a multi-layer model. To solve this problem, we use the straightforward technique of masking out some of the words in the input and then condition each word bidirectionally to predict the masked words."

Can someone please help me understand how does bidirectionally allow the words to see themselves and how masking solves this problem?

Thanks.

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
2
$\begingroup$

Let's take an example : "I went to the shop."

Let's say you want to predict "to" and "the". With bidirectionality, you will predict :

p(to | I, went, the, shop) : No problem here.

p(the | I, went, to, shop) : Here we have a problem, because we already saw the word 'the' while predicting 'to'. It's trivial for the model to predict 'the'.

With MLM, if we take the same example :

p(to | I, went, [MASK], [MASK], shop)
p(the | I, went, [MASK], [MASK], shop)

There is no more problem because the word cannot see itself in other predictions. It's more difficult to predict, but anyway only 15% of the time the word is masked.

Edit

In MLM, BERT will see all the words of the sentence, including the word to predict itself. This is because BERT will create representations not only for the [MASK] token, but also for the other tokens of the sentence (but in this case we are interested only in [MASK]).

Note : It is also because BERT will create representations of all tokens that we need to replace [MASK] sometimes by the original token, sometimes by a random token.

You can find more detailed information in this blog in the "Pretraining" section.

Improve this answer
$\endgroup$
2
  • $\begingroup$ With MLM, shouldn't the probability be P(to | I,went, [MASK],shop ) with only one [MASK] ? $\endgroup$
    – Zephyr
    Commented Aug 6, 2019 at 4:13
  • $\begingroup$ Please see my edit :) $\endgroup$
    – Astariul
    Commented Aug 6, 2019 at 4:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.