I am doing some study about the BatchNormalization: https://towardsdatascience.com/batch-normalization-8a2e585775c9

In the article, it says:

Using batch normalization allows us to use much higher learning rates, which further increases the speed at which networks train.

Could anyone please share their thoughts on why would batch normalization allow higher learning rate? Thanks!


1 Answer 1


Section 3.3 of the original batch normalization paper has a good explanation of why this is the case.

Problem with a higher learning rate

First you need to understand the problem with higher learning rate. Higher learning rate causes exploding or vanishing gradients. In other words, gradients get multiplied by each other, so lower layers experience a compound effect of the gradients that are in higher levels.

How does batch normalization help?

Batch normalization is all about keeping the activations of all layers normalized, preventing them from becoming too large or small. So this directly helps to prevent exploding/vanishing gradient. Due to this reason, batch normalization allows higher learning rates.

  • 1
    $\begingroup$ I don't get why keeping all activations normalized helps in vanishing or exploding gradients. Can you elaborate on this? $\endgroup$
    – ado sar
    Commented Feb 10 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.