what's the motivation behind BERT masking 2 words in a sentence?

bert and the more recent t5 ablation study, agree that

using a denoising objective always results in better downstream task performance compared to a language model

where denoising == masked-lm == cloze.

I understand why learning to represent a word according to its bidirectional surroundings makes sense. However, I fail to understand why is it beneficial to learn to mask 2 words in the same sentence, e.g. The animal crossed the road => The [mask] crossed the [mask]. Why does it make sense to learn to represent animal without the context of road?

Note: I understand that the masking probability is 15% which corresponds to 1/7 words, which makes it pretty rare for 2 words in the same sentence to be masked, but why would it ever be beneficial, even with low probability?

Note2: please ignore the masking procedure sometimes replacing mask with a random/same word instead of [mask], T5 investigates this choice in considerable length and I suspect that it's just an empirical finding :)

• thanks! but i don't understand why is independence making the problem simpler as you state? won't it be a lot simpler if we had just one masked token (which is the only predicted token) per sentence? Jan 25 at 18:47