Is there any concern for a pretrained model to overfitting to a fine-tuning task that has overlapping pretraining and training data?

Question

Let's say my language model is pretrained on a general text corpus, and I want to use it for some specific downstream task that has it's datasets also included in the general corpus, is there any concern for overfitting or bias?

I can't seem to find much resources that touch on this issue. I read this paper SciBERT that shows in-domain pretraining of BERT with vocab and corpus extracted from only scientific text would yield better performance on scientific tasks. But isn't this just overfitting? I also read a few papers like the T5 paper that claims in-domain pretraining leads to improvement of fine-tuning tasks as if it is a merit to use pretraining data that is similar to finetuning tasks? Is there not a concern for overfitting? Is it not a concern if the pretraining and finetuning objectives are different enough? Or am I misunderstanding the concept of pretraining and overfitting?

Would appreciate if anyone could also provide links to articles that investigate this issue.

Answer 1

You are mixing to unrelated concepts. Pretraining/finetuning and overfitting are not related.

First, let's clarify some concepts:

• Overfitting: from wikipedia:

  "the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit to additional data or predict future observations reliably"

  This means that overfitting is when our model learns "too well" the training data and, when faced with data at inference, it performs poorly.

• Test set: it is a set of data that is not a subset of the training data but that is draw from the same distribution as the training data. It helps us assess the model's performance after training. It is important that there is no overlap between the test and the training data, to ensure that the evaluation is faithful; otherwise, when faced with real inference data has not actually been seen during training, the model performance would probably not match its performance on the test set.

• Pretraining: it is when we train a model on a large training data, so that we can do fine tuning over a not-so-large data from our downstream task. The distributions of pretraining data and the training data are by definition different, although similar to some degree. Of course, the more similar they are, the more useful would be the pretrained model. Some examples of pretraining and finetuning data are in textual data:

  • General domain text → Domain-specific text
  • Multiple language text → Single language text
  • Text in one language (e.g. Spanish) → text in a similar language (e.g. Portuguese)

Now, to answer your question: having an overlap between the pretraining data and the finetuning data is not related to overfitting, because overfitting refers to the trained model having different behaviour when applied to the training data (performs good) and the test data (performs bad). As long as the downstream task test data is not leaked to the pretraining or the finetuning training sets, there should be no problem.

Using a pretrained model is just a means to having a "starting point" in the model optimization that is assumed to be better than random initialization. The more similar the pretraining data to the finetuning data, the better such an "starting point" would be and, presumably the better the end performance on the downstream task.