0
$\begingroup$

A blog below mentioned.

" Because different features do not have similar ranges of values, gradients may take a long time, oscillate back and forth, and take a long time before they can finally find their way to the global/local minimum."

Can someone explain clearly, why do we have problems with gradients when feature values are of different ranges?

$\endgroup$

1 Answer 1

2
$\begingroup$

To understand the reason behind this, imagine you are going to make a prediction using regression with two predictors $x_1$ and $x_2$ and the coefficients are $\theta_1$ and $\theta_2$.

If we plot $\theta_1$, $\theta_2$, and the cost function:

enter image description here

You can see that the cost function is stretched over the axis of the smaller scale feature like the right one. But after scaling the features you will have a cost function like the left one.

Now note that more stretching of the function in one direction causes smaller gradients so more steps are needed to reach the minimum. Thus without feature scaling the gradient descent algorithm may take a much longer time to converge.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.