The masked language model task is the key to BERT and RoBERTa. However, they differ in how they prepare such masking. The original RoBERTa article explains it in section 4.1:
BERT relies on randomly masking and predicting tokens. The original BERT implementation performed masking
once during data preprocessing, resulting in a single static mask. To avoid using the same mask for
each training instance in every epoch, training data
was duplicated 10 times so that each sequence is
masked in 10 different ways over the 40 epochs of
training. Thus, each training sequence was seen
with the same mask four times during training.
We compare this strategy with dynamic masking where we generate the masking pattern every
time we feed a sequence to the model. This becomes crucial when pretraining for more steps or
with larger datasets.
This way, in BERT, the masking is performed only once at data preparation time, and they basically take each sentence and mask it in 10 different ways. Therefore, at training time, the model will only see those 10 variations of each sentence.
On the other hand, in RoBERTa, the masking is done during training. Therefore, each time a sentence is incorporated in a minibatch, it gets its masking done, and therefore the number of potentially different masked versions of each sentence is not bounded like in BERT.