I'm using a ResNet50 model pretrained on ImageNet, to do transfer learning, fitting an image classification task.
The easy way of doing this is simply freezing the conv layers (or really all layers except the final fully connected layer), however I came across a paper where the authors mention that batch normalisation layers should be fine tuned when fitting the new model:
Few layers such as Batch Normalization (BN) layers shouldn’t be froze because, the mean and variance of the dataset will be hardly matching the mean or variance from pre-trained weights. So, auto-tuning  is adapted for the BN layers in ResNey50, i.e. few of the layers which are in the top of ResNet50 shouldn’t be frozen.
I initially froze everything, but in the interest of time and not having to commit computation to testing the effect of not freezing the BN layers, how important is it to not freeze them?