One question about the pre-training & fine-tuning process for language models: why is it better to fine-tune using a small dataset rather than including the fine-tuning dataset into the pre-training dataset?

Or am I misunderstanding and normally the fine-tuning dataset is already included in the pre-training one, and we only change the learning parameters to better fit the data properties?

Any paper reference is very welcome! Thank you.


The main reason is that, normally, the tasks you pretrain and finetune the model are different, e.g. masked language modeling vs. sequence classification or tagging. This is because unlabeled data is abundant and labeled data is scarce, and pretraining on a language modeling task allows you to use a lot of data in a scarce data setup. This is, for instance, commented in the ULMfit paper, which was one of the ones that started the fine-tuning LMs wave:

While we have shown that ULMFiT can achieve state-of-the-art performance on widely used text classification tasks, we believe that language model fine-tuning will be particularly useful in the following settings compared to existing transfer learning approaches (Conneau et al., 2017; McCann et al., 2017; Peters et al., 2018): a) NLP for non-English languages, where training data for supervised pretraining tasks is scarce; b) new NLP tasks where no state-of-the-art architecture exists; and c) tasks with limited amounts of labeled data (and some amounts of unlabeled data).

Also, we apply transfer learning (first pretrain, then finetune) and not multitask learning (train both losses together), because what we want is to have good performance in the downstream task, and we don't care so much about the pretraining task apart from being a means to improve our results in the downstream one.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.