Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
From this article, it says:
We can speed up gradient descent by scaling. This is because θ will descend quickly on small ranges and slowly on large ranges, and so will oscillate inefficiently down to the optimum when the variables are very uneven.
For linear regresion, from the equation $a_{n+1}=a_n-\alpha\nabla F(a_n)$ it wasn't obvious to me why larger range variables converge slower or how/why an oscillation occurs.
The answer to this question simply lies in how one performs or applies the chain rule of differentiation. To understand this, first look at back-propagation which is simply a method to update weights by calculating the error between the predicted value and the exact value. When this error is back-propagated to optimize your objective function, derivatives are calculated in the subsequent layers and are multiplied by learning rate and the weights are then updated.
Now just imagine, if you have some features and range of one feature is from 0 to 1 and the other feature's range is from 10 to 1000. Now when you do differentiation for back-propagation for features with high range values you may get derivatives which are greater as compared to features which have low range values, this would lead to abnormal weight updating, and would add certain biasness to our model regarding those features, even when that feature as compared to the feature which has low range values may hold equal importance when statistical testing is done.
Thus to avoid this, introduction of biasness, feature scaling is used which allows us to scale features in a standard scale without associating any kind of biasness to it. This applies to various machine learning models such as SVM, KNN etc as well as neural networks. Another reason to scale features is to avoid large derivatives being back-propagated which increases the computation.
Another curious aspect of scaling is if you don't scale your features you will have different error surface (because of introduction of biasness when it is not there) as compared to scaled features error surface this may prevent you from reaching the optimum minimum value for your objective function thus that is why scaling helps you to ensure that you at least reach local minima thus it can be said scaling helps you to increase convergence speed while doing gradient descent.
Centring the data makes the error function more spherical which in turn helps gradient descent to converge faster as otherwise, the loss function may have a higher curvature and require more steps to converge. https://stats.stackexchange.com/questions/315027/why-is-mean-normalization-useful-in-gradient-descent
