I would like to fine-tune a pre-trained BERT-like model for a semantic similarity analysis task in the fashion of the SNLI/MNLI task (i.e. classify sentence pairs to "entailment" or "contradiction"). I have my own domain specific dataset but it is very small. For that reason I planned to fine-tune the model first on a large "general" dataset like the MNLI and then further fine-tune it on my own data. This way I want to take advantage of the large publicly available dataset and thus, hopefully improve results on my small dataset.
My questions here are:
- Does this approach make sense in general for language models? Or would phenomena like catastrophic forgetting break my plan anyway?
- How do I design the training steps? Should I freeze some of the (additional) fine-tuned layers (apart from the embedding layers of course) or do I simply resume training from last checkpoint?