# In Gradient descent, Why the gradient of cost function do not have to be normalized into unit vector

From my background, I understand that the purpose of having a learning rate (α) is to normalize the magnitude of gradient (▽J), so the step size can properly converge the local minima

Since α is arbitrary, so we have to do find the best hyperparameter learning rate (α).

Perhap, we do not have to hyper-parameter tuning when we normalize the gradient into a unit vector in which its magnitude always 1

w:=w −α▽J


w:=w −▽J/||▽J||


Again, back to my problem, I do not understand why the variant of gradient descent does not have what I think. Like normalize the gradient into unit vector

ps. I presume that my Gradient Descent might not work well when the magnitude of gradient is extremely low and could diverge to inf

In a gradient descent algorithm, the algorithm proceeds by finding a direction along which you can find the optimal solution. The optimal direction turns out to be the gradient. However, since we are only interested in the direction and not necessarily how far we move along that direction, we are usually not interested in the magnitude of the gradient. A normalized gradient is good enough for our purposes, and we let 𝜂 dictate how far we want to move in the computed direction.

There is no difference between normalized and unnormalized gradient descent (as far as the theory behind the algorithm goes). However, it has a practical impact on the speed of convergence and stability. The choice of one over the other is purely based on the application/objective at hand.

The most obvious reason is that a gradient of the norm of 1 is expected to be at learning_rate * 1 away from the loss function minimum. That is averaged of course.
Also, think about what happens when $$\nabla J \to 0$$. What would $$\frac{\nabla J}{|\nabla J|}$$ tend to? And the effect of this limit on convergence without an appropriate learning rate?