I am reading the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding that can be found here.
It looks to me that the crux of the paper is using masked inputs to achieve bidirectionally.
This is an excerpt from the Google AI blog here which states:
"However, it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that’s being predicted to indirectly “see itself” in a multi-layer model. To solve this problem, we use the straightforward technique of masking out some of the words in the input and then condition each word bidirectionally to predict the masked words."
Can someone please help me understand how does bidirectionally allow the words to see themselves and how masking solves this problem?