I want to understand the difference between BERT and Roberta. I saw the article below.
https://towardsdatascience.com/bert-roberta-distilbert-xlnet-which-one-to-use-3d5ab82ba5f8
It mentions that Roberta was trained on 10x more data but I don't understand the dynamic masking part. It says masked tokens change between epochs. Shouldn't this flatten the learning curve?
The masked language model task is the key to BERT and RoBERTa. However, they differ in how they prepare such masking. The original RoBERTa article explains it in section 4.1:
BERT relies on randomly masking and predicting tokens. The original BERT implementation performed masking once during data preprocessing, resulting in a single static mask. To avoid using the same mask for each training instance in every epoch, training data was duplicated 10 times so that each sequence is masked in 10 different ways over the 40 epochs of training. Thus, each training sequence was seen with the same mask four times during training.
We compare this strategy with dynamic masking where we generate the masking pattern every time we feed a sequence to the model. This becomes crucial when pretraining for more steps or with larger datasets.
This way, in BERT, the masking is performed only once at data preparation time, and they basically take each sentence and mask it in 10 different ways. Therefore, at training time, the model will only see those 10 variations of each sentence.
On the other hand, in RoBERTa, the masking is done during training. Therefore, each time a sentence is incorporated in a minibatch, it gets its masking done, and therefore the number of potentially different masked versions of each sentence is not bounded like in BERT.
The dynamic masking is analogous to using different image augmentations so you can reuse the same image for training repeatedly but the network actually sees a different example.
Concretely, imagine we were training a network to perform in-painting. For training we have a complete image and then choose some region to occlude and ask the network to predict what the occluded part is supposed to look like.
Now imagine that each epoch we reuse this image but change the location of the occlusion. There's a bit of data leakage risk here (we're asking the network to predict a region it's actually seen before), but with an appropriately large dataset that shouldn't be a problem: the network will perform better on the rest of the dataset if it learns generally useful features than if it memorizes that one image. Memorization isn't necessarily even a bad thing if the network learns how to mix and match what it memorized in clever ways (i.e. treating an image as a bag of local features).
RoBERTa's dynamic masking is just the text version of that. Instead of an image, we have a chunk of text. Instead of occluding a region of pixels, we're occluding a region of text. It's a data augmentation that functionally increases the variability of the data, encouraging the network to learn more robust features.
