Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am doing some study about the BatchNormalization: https://towardsdatascience.com/batch-normalization-8a2e585775c9
In the article, it says:
Using batch normalization allows us to use much higher learning rates, which further increases the speed at which networks train.
Could anyone please share their thoughts on why would batch normalization allow higher learning rate? Thanks!
Section 3.3 of the original batch normalization paper has a good explanation of why this is the case.
First you need to understand the problem with higher learning rate. Higher learning rate causes exploding or vanishing gradients. In other words, gradients get multiplied by each other, so lower layers experience a compound effect of the gradients that are in higher levels.
Batch normalization is all about keeping the activations of all layers normalized, preventing them from becoming too large or small. So this directly helps to prevent exploding/vanishing gradient. Due to this reason, batch normalization allows higher learning rates.
