Let's say my language model is pretrained on a general text corpus, and I want to use it for some specific downstream task that has it's datasets also included in the general corpus, is there any concern for overfitting or bias?

I can't seem to find much resources that touch on this issue. I read this paper SciBERT that shows in-domain pretraining of BERT with vocab and corpus extracted from only scientific text would yield better performance on scientific tasks. But isn't this just overfitting? I also read a few papers like the T5 paper that claims in-domain pretraining leads to improvement of fine-tuning tasks as if it is a merit to use pretraining data that is similar to finetuning tasks? Is there not a concern for overfitting? Is it not a concern if the pretraining and finetuning objectives are different enough? Or am I misunderstanding the concept of pretraining and overfitting?

Would appreciate if anyone could also provide links to articles that investigate this issue.

  • $\begingroup$ Do you find any references for this ? $\endgroup$ Oct 7, 2023 at 2:29
  • $\begingroup$ You are asking: if I have a pretrained model and I have some task head which solves for a task and in the test set if the task head some data occurs which was seen during pretraining: is this a problem, i.e. making the test set performance look better than it is on real/completely held out data? Yes, I would be concerned! People talked about this when discussing the performance of GPT4 $\endgroup$
    – Ggjj11
    Oct 12, 2023 at 17:50

1 Answer 1


You are mixing to unrelated concepts. Pretraining/finetuning and overfitting are not related.

First, let's clarify some concepts:

  • Overfitting: from wikipedia:

    "the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit to additional data or predict future observations reliably"

    This means that overfitting is when our model learns "too well" the training data and, when faced with data at inference, it performs poorly.

  • Test set: it is a set of data that is not a subset of the training data but that is draw from the same distribution as the training data. It helps us assess the model's performance after training. It is important that there is no overlap between the test and the training data, to ensure that the evaluation is faithful; otherwise, when faced with real inference data has not actually been seen during training, the model performance would probably not match its performance on the test set.

  • Pretraining: it is when we train a model on a large training data, so that we can do fine tuning over a not-so-large data from our downstream task. The distributions of pretraining data and the training data are by definition different, although similar to some degree. Of course, the more similar they are, the more useful would be the pretrained model. Some examples of pretraining and finetuning data are in textual data:

    • General domain text → Domain-specific text
    • Multiple language text → Single language text
    • Text in one language (e.g. Spanish) → text in a similar language (e.g. Portuguese)

Now, to answer your question: having an overlap between the pretraining data and the finetuning data is not related to overfitting, because overfitting refers to the trained model having different behaviour when applied to the training data (performs good) and the test data (performs bad). As long as the downstream task test data is not leaked to the pretraining or the finetuning training sets, there should be no problem.

Using a pretrained model is just a means to having a "starting point" in the model optimization that is assumed to be better than random initialization. The more similar the pretraining data to the finetuning data, the better such an "starting point" would be and, presumably the better the end performance on the downstream task.

  • $\begingroup$ I guess the OP is worried that if the model sees the same documents too much times it will overfit to them. This is why LLM are trained only for 1 or 2 epochs, because they quickly overfit the training data. $\endgroup$
    – alexmolas
    Oct 10, 2023 at 8:51
  • 1
    $\begingroup$ @alexmolas the overfitting that is addressed by finetuning only a couple of epochs is actually caused not so much from an overlap between pretraining and finetuning datasets, but from the small size of the finetuning dataset, don't you agree? 1/2 $\endgroup$
    – noe
    Oct 10, 2023 at 17:17
  • $\begingroup$ Pretraining datasets are huge, that's why they are useful for learning good representations. I understand that a finetuning dataset that was originally part of the pretraining set and just comprised a tiny fraction of it, is unlikely to lead to an overfitting problem because of the overlap. If the finetuning dataset comprised a large fraction of the pretraining dataset, it is likewise unlikely to lead to an overfitting problem due to its big size. 2/2 $\endgroup$
    – noe
    Oct 10, 2023 at 17:17
  • $\begingroup$ @noe Would you kindly have a look to my question? $\endgroup$
    – Mario
    Oct 10, 2023 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.