I would like to fine-tune a pre-trained BERT-like model for a semantic similarity analysis task in the fashion of the SNLI/MNLI task (i.e. classify sentence pairs to "entailment" or "contradiction"). I have my own domain specific dataset but it is very small. For that reason I planned to fine-tune the model first on a large "general" dataset like the MNLI and then further fine-tune it on my own data. This way I want to take advantage of the large publicly available dataset and thus, hopefully improve results on my small dataset.

My questions here are:

  1. Does this approach make sense in general for language models? Or would phenomena like catastrophic forgetting break my plan anyway?
  2. How do I design the training steps? Should I freeze some of the (additional) fine-tuned layers (apart from the embedding layers of course) or do I simply resume training from last checkpoint?

1 Answer 1


When it comes to hyperparameter tuning, automatic tuners such as keras tuner is the way to go. Specially consider using Bayesian optimization, since it can relatively outperform the other methods such as Grid and Radnom search. PS: Be sure to set the initial random points of the Bayesian Optimizer very high (at least 30, even higher if you have a really big hyperspace) to ensure that Bayesian Optimizer has enough prior data to build the surrogate model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.