# Batch normalization vs batch size

I have noticed that my performance of VGG 16 network gets better if I increase the batch size from $$64$$ to $$256$$. I have also observed that, using batch size $$64$$, the with and without batch normalization results have lot of difference. With batch norm results being poorer. As I increase the batch size the performance of with and without batch normalization gets closer. Something funky going on here.

So I would like to ask is the increase in batch size have effect on batch normalization?

• Adding batchnorm right after the relu layers? Nov 30, 2018 at 6:57
• tried both after relu and before relu, both results are poorer than without BN. Nov 30, 2018 at 15:39
• In Pytorch, for their faster R-CNN Resnet 50 fpn architecture in torchvision, I've seen it actually recommended to disable batch normalization since the images are 800 by 800 by 3 tensors, and even a batch of 8 with 16 bit floating precision will be too large for a lot of entry level ML GPUs. Feb 4, 2021 at 6:58

By increasing batch size your steps can be more accurate because your sampling will be closer to the real population. If you increase the size of batch, your batch normalisation can have better results. The reason is exactly like the input layer. The samples will be closer to the population for inner activations.

• What you wrote, completely makes sense to me. Thanks. Nov 30, 2018 at 15:40

While it's true that increasing the batch size will make the batch normalization stats (mean, variance) closer to the real population, and will also make gradient estimates closer to the gradients computed over the whole population allowing the training to be more stable (less stochastic), it is necessary to note that there is a reason why we don't use the biggest batch sizes we can computationally afford.

Let's say that our hardware configuration allows us to train using batches of 10K samples. Sure, our BN would get more precise stats, and our gradients would be more accurate and closer to the real gradient. However, this is not necessarily good because mini-batch SGD is proven to work better when the batches aren't big. As batches get remarkably bigger, mini-batch SGD becomes more and more like its father gradient descent, and this ancient monster isn't good for non-convex optimization problems like deep neural networks for both computational reasons and local-minima-related reasons.

• Since ultimately even the training examples are a subset of all possible true examples in the general set of inputs, is it not true that even a monster update on the whole training set is still in a sense a kind of mini-batch SGD. I'd be interested to see when for object detection/instance segmentation models in CV that batch sizes could be "too big" with the appropriate learning rate. Feb 4, 2021 at 7:01