To freeze or not, batch normalisation in ResNet when transfer learning

Question

I'm using a ResNet50 model pretrained on ImageNet, to do transfer learning, fitting an image classification task.

The easy way of doing this is simply freezing the conv layers (or really all layers except the final fully connected layer), however I came across a paper where the authors mention that batch normalisation layers should be fine tuned when fitting the new model:

Few layers such as Batch Normalization (BN) layers shouldn’t be froze because, the mean and variance of the dataset will be hardly matching the mean or variance from pre-trained weights. So, auto-tuning [30] is adapted for the BN layers in ResNey50, i.e. few of the layers which are in the top of ResNet50 shouldn’t be frozen.

source here

I initially froze everything, but in the interest of time and not having to commit computation to testing the effect of not freezing the BN layers, how important is it to not freeze them?

Answer 1

The batch normalization layer helps with effectively training the model. Since you are transfer learning, you may have frozen everything up to the fully connected classifier. If you want to keep the parameters of the frozen layers exactly the same as the original model, you can load the weights of only the retrained head during inference and/or evaluation.

Your best bet is to try everything and see what works best for you.