2
$\begingroup$

In the ULMFit paper authors propose a strategy of gradual unfreezing in order to deal with catastrophic forgetting. That is, when the model starts be fine-tuned according to a downstream task, there is the danger of forgetting information on lower layers. Although Google's BERT is also a pre-trained language model, which makes use of fine-tuning for downstream tasks, authors don't mention this phenomenon. Why is that the case? Is BERT immune to it? Or does it deal with this in another way?

$\endgroup$
0
$\begingroup$

As far as I know, no neural network can be immune to catastrophic forgetting during fine-tuning (which is essentially controlled retraining).

The key is to not fine tune the pretrained model for longer epochs, or with higher learning rates. This ensures that the learned knowledge from the lower layers is preserved more or less intact, while also helping the model learn from the new data used for fine-tuning, as mentioned here : https://github.com/huggingface/transformers/issues/1019

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.