11
$\begingroup$

I have noticed that my performance of VGG 16 network gets better if I increase the batch size from $64$ to $256$. I have also observed that, using batch size $64$, the with and without batch normalization results have lot of difference. With batch norm results being poorer. As I increase the batch size the performance of with and without batch normalization gets closer. Something funky going on here.

So I would like to ask is the increase in batch size have effect on batch normalization?

Improve this question
$\endgroup$
3
  • $\begingroup$ Adding batchnorm right after the relu layers? $\endgroup$
    – Aditya
    Commented Nov 30, 2018 at 6:57
  • $\begingroup$ tried both after relu and before relu, both results are poorer than without BN. $\endgroup$
    – Arka Mallick
    Commented Nov 30, 2018 at 15:39
  • $\begingroup$ In Pytorch, for their faster R-CNN Resnet 50 fpn architecture in torchvision, I've seen it actually recommended to disable batch normalization since the images are 800 by 800 by 3 tensors, and even a batch of 8 with 16 bit floating precision will be too large for a lot of entry level ML GPUs. $\endgroup$
    – IntegrateThis
    Commented Feb 4, 2021 at 6:58

2 Answers 2

Reset to default
9
$\begingroup$

By increasing batch size your steps can be more accurate because your sampling will be closer to the real population. If you increase the size of batch, your batch normalisation can have better results. The reason is exactly like the input layer. The samples will be closer to the population for inner activations.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ What you wrote, completely makes sense to me. Thanks. $\endgroup$
    – Arka Mallick
    Commented Nov 30, 2018 at 15:40
7
$\begingroup$

While it's true that increasing the batch size will make the batch normalization stats (mean, variance) closer to the real population, and will also make gradient estimates closer to the gradients computed over the whole population allowing the training to be more stable (less stochastic), it is necessary to note that there is a reason why we don't use the biggest batch sizes we can computationally afford.

Let's say that our hardware configuration allows us to train using batches of 10K samples. Sure, our BN would get more precise stats, and our gradients would be more accurate and closer to the real gradient. However, this is not necessarily good because mini-batch SGD is proven to work better when the batches aren't big. As batches get remarkably bigger, mini-batch SGD becomes more and more like its father gradient descent, and this ancient monster isn't good for non-convex optimization problems like deep neural networks for both computational reasons and local-minima-related reasons.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Since ultimately even the training examples are a subset of all possible true examples in the general set of inputs, is it not true that even a monster update on the whole training set is still in a sense a kind of mini-batch SGD. I'd be interested to see when for object detection/instance segmentation models in CV that batch sizes could be "too big" with the appropriate learning rate. $\endgroup$
    – IntegrateThis
    Commented Feb 4, 2021 at 7:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.