So I was reading this paper (about a use case of pretraining then self-training) which got me thinking - suppose I pre-train a model on a particular dataset, then fine-tune it again on the same dataset.

Theoretically, if we pre-train it as a masked LM and fine-tune, it might lead to overfitting - but I am not sure. Maybe it won't be able to generalize well, but would still give an accuracy boost?

Does anyone know some research or other credible sources to explain why or why not this should be done?

Also, if this is indeed possible/advised do we have to take some certain extra steps in Tensorflow, Keras, or HuggingFace (basically any Deep-Learning Framework or library) for doing so?

EDIT:- A simple example, I am saying like if we have a specific supervised task, and pre-train on the same features as we would fine-tune on - would that increase accuracy by any bit?


1 Answer 1


In the context of that paper, pre-train then fine-tune on the same dataset does not really make sense, as the pre-training is unsupervised, and the fine-tuning is with labelled data.

But, generally, if you have trained on dataset X for N epochs, and then you fine-tune one more epoch using the whole of X, it is just another way of saying you trained for N+1 epochs. Nothing wrong with it, except as you note in the question, if you were starting to overfit after N epochs, you are even more over-fitted now.

What does make sense, and something we have used in production models, is to do the initial training on a large dataset that is the combination of X1, X2, X3, ... and so on. Then once the learning curve starts to level out, we take a copy of the model and then fine-tune on e.g. just X1.

This is in the context of NLP Transformer models, so may not make sense for other domains, but we try to do initial training on as much data, from all domains, as possible, then we might fine-tune on just medical papers, or just economic reports, depending on what the model will be used for. (We've also done a final fine-tune on just a subset of the data to have it learn a specific style. It is amazing that even with just one epoch of this final fine-tune it can switch to using a different way of writing numbers, for instance.)

  • $\begingroup$ alright, so hear me - if suppose I am doing sentiment classification and I concatenate the unlabelled test set with unlabelled train set, pre-train on it and then fine-tune on the labelled train set, would it give a higher accuracy or not? $\endgroup$
    – neel g
    Apr 19, 2021 at 20:30
  • 1
    $\begingroup$ @neelg Definitely maybe. It depends what it could learn from the unlabelled test set that it couldn't learn from the unlabelled train set. And I'd want to think carefully about what kind of things it could possibly learn before deciding if it is compromising the usefulness of my test set. E.g. I'm happy to include my test set in the data I create the sentencepiece model from; the only way it could matter is if the train and test data came from notably different distributions. $\endgroup$ Apr 19, 2021 at 21:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.