I have recently read about Fine Tuning, and what I want to know is, when we are fine-tuning our model is it necessary to Freeze the model and train only the top part of the model and then unfreeze some layers and again train the model or one can directly begin by unfreezing some layers? Till now, I have read that one does not unfreeze the layers directly because then we risk losing the important features captured by the earlier layers
It can work either way. If you want to keep the exact feature extractors, then you should freeze everything except the "top" of the model. You can also unfreeze the whole model; the "top" of the model will be trained from scratch, and the feature extractors near the "bottom" of the model will be tweaked to work better with your dataset. The potential drawback of unfreezing the whole model is a higher potential for overfitting (and a longer, more expensive training time)
[I]s it necessary to Freeze the model and train only the top part of the model and then unfreeze some layers and again train the model or one can directly begin by unfreezing some layers?
I'm not aware of any training routine that involves freezing and unfreezing different parts of the model at different times during training. People may have done this, but I'm not sure what the benefits would be.
It is a good practice to freeze the early layers when you finetune the model. There are two reasons why you want to do this :
- Your new layers is initialized randomly and will always start with very big loss. Big loss means big gradient and if the weights are not frozen this will be propagated . Your model will become less stable and might fail to converge. To put it simply you don't want to put the blame of false prediction on the earlier layers.
- You are saving a lot of computation resources. Training a neural network, is not a small issue, and training only a small section of the model will give you decent enough accuracy without expending too much computational resource.
Unfreezing the earlier layer is up to you on the later stage of training (Some people do some people don't) but if you feel you might need to do this to push for better performance, then just try it.