BERT Masked Language Model question

Question

I have been reading about BERT from the internet, and from what I understand the point of masked language modelling for BERT pretraining is so that BERT will learn to guess a "masked" word from the context given. The loss function will be the lowest for output embeddings which are closest to the original masked word embedding. Wouldn't it be that using this loss funtion does not guarantee that BERT will output word embeddings with context and instead could just output an embedding closest to the original masked word embedding without the relevant context?

Thanks.

Answer 1

In this context, masking means replacing the token with a special [MASK] token. The network does not have the information of what the original token was, the only way how it could potentially figure out what it was is by looking at the context.

It is not the loss function that guarantees that the model learns something meaningful, it is architecture design that does not allow the workaround that you suggest.