So I was reading this paper (about a use case of pretraining then self-training) which got me thinking - suppose I pre-train a model on a particular dataset, then fine-tune it again on the same dataset.
Theoretically, if we pre-train it as a masked LM and fine-tune, it might lead to overfitting - but I am not sure. Maybe it won't be able to generalize well, but would still give an accuracy boost?
Does anyone know some research or other credible sources to explain why or why not this should be done?
Also, if this is indeed possible/advised do we have to take some certain extra steps in Tensorflow, Keras, or HuggingFace (basically any Deep-Learning Framework or library) for doing so?
EDIT:- A simple example, I am saying like if we have a specific supervised task, and pre-train on the same features as we would fine-tune on - would that increase accuracy by any bit?