using a denoising objective always results in better downstream task performance compared to a language model
where denoising == masked-lm == cloze.
I understand why learning to represent a word according to its bidirectional surroundings makes sense. However, I fail to understand why is it beneficial to learn to mask 2 words in the same sentence, e.g.
The animal crossed the road =>
The [mask] crossed the [mask]. Why does it make sense to learn to represent
animal without the context of
Note: I understand that the masking probability is 15% which corresponds to 1/7 words, which makes it pretty rare for 2 words in the same sentence to be masked, but why would it ever be beneficial, even with low probability?
Note2: please ignore the masking procedure sometimes replacing mask with a random/same word instead of
[mask], T5 investigates this choice in considerable length and I suspect that it's just an empirical finding :)