In Gradient descent, Why the gradient of cost function do not have to be normalized into unit vector

Question

From my background, I understand that the purpose of having a learning rate (α) is to normalize the magnitude of gradient (▽J), so the step size can properly converge the local minima

Since α is arbitrary, so we have to do find the best hyperparameter learning rate (α).

Perhap, we do not have to hyper-parameter tuning when we normalize the gradient into a unit vector in which its magnitude always 1

The traditional gradient descent:

w:=w −α▽J

My gradient descent:

w:=w −▽J/||▽J||

Again, back to my problem, I do not understand why the variant of gradient descent does not have what I think. Like normalize the gradient into unit vector

ps. I presume that my Gradient Descent might not work well when the magnitude of gradient is extremely low and could diverge to inf

Answer 1

In a gradient descent algorithm, the algorithm proceeds by finding a direction along which you can find the optimal solution. The optimal direction turns out to be the gradient. However, since we are only interested in the direction and not necessarily how far we move along that direction, we are usually not interested in the magnitude of the gradient. A normalized gradient is good enough for our purposes, and we let 𝜂 dictate how far we want to move in the computed direction.

There is no difference between normalized and unnormalized gradient descent (as far as the theory behind the algorithm goes). However, it has a practical impact on the speed of convergence and stability. The choice of one over the other is purely based on the application/objective at hand.

I think this has already been answered here.

Answer 2

The most obvious reason is that a gradient of the norm of 1 is expected to be at learning_rate * 1 away from the loss function minimum. That is averaged of course.

Proper gradient near the minimum approaches to 0 due to cancellation of the first order term in Taylor expansion of Loss function around the minimum. So at least in theory, a step based on a Loss gradient rather than normalised gradient can converge towards the minimum.

Answer 3

Your intuition is partly correct, and in fact the family of adaptive learning rate methods (adagrad, adadelta, adam, etc) takes into consideration the magnitude of the gradient, quoting from ADADELTA: AN ADAPTIVE LEARNING RATE METHOD

Since the magnitudes of gradients are factored out in ADAGRAD, this method can be sensitive to initial conditions of the parameters and the corresponding gradient.

However, the paper also points out a problem in doing so.
Also, think about what happens when $\nabla J \to 0$. What would $\frac{\nabla J}{|\nabla J|} $ tend to? And the effect of this limit on convergence without an appropriate learning rate?