I have noticed that my performance of VGG 16 network gets better if I increase the batch size from $64$ to $256$. I have also observed that, using batch size $64$, the with and without batch normalization results have lot of difference. With batch norm results being poorer. As I increase the batch size the performance of with and without batch normalization gets closer. Something funky going on here.

So I would like to ask is the increase in batch size have effect on batch normalization?

  • $\begingroup$ Adding batchnorm right after the relu layers? $\endgroup$ – Aditya Nov 30 '18 at 6:57
  • $\begingroup$ tried both after relu and before relu, both results are poorer than without BN. $\endgroup$ – Haramoz Nov 30 '18 at 15:39

By increasing batch size your steps can be more accurate because your sampling will be closer to the real population. If you increase the size of batch, your batch normalisation can have better results. The reason is exactly like the input layer. The samples will be closer to the population for inner activations.

  • 1
    $\begingroup$ What you wrote, completely makes sense to me. Thanks. $\endgroup$ – Haramoz Nov 30 '18 at 15:40

While it's true that increasing the batch size will make the batch normalization stats (mean, variance) closer to the real population, and will also make gradient estimates closer to the gradients computed over the whole population allowing the training to be more stable (less stochastic), it is necessary to note that there is a reason why we don't use the biggest batch sizes we can computationally afford.

Let's say that our hardware configuration allows us to train using batches of 10K samples. Sure, our BN would get more precise stats, and our gradients would be more accurate and closer to the real gradient. However, this is not necessarily good because mini-batch SGD is proven to work better when the batches aren't big. As batches get remarkably bigger, mini-batch SGD becomes more and more like its father gradient descent, and this ancient monster isn't good for non-convex optimization problems like deep neural networks for both computational reasons and local-minima-related reasons.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.