1
$\begingroup$

From my background, I understand that the purpose of having a learning rate (α) is to normalize the magnitude of gradient (▽J), so the step size can properly converge the local minima

Since α is arbitrary, so we have to do find the best hyperparameter learning rate (α).

Perhap, we do not have to hyper-parameter tuning when we normalize the gradient into a unit vector in which its magnitude always 1

The traditional gradient descent:

w:=w −α▽J

My gradient descent:

w:=w −▽J/||▽J||

Again, back to my problem, I do not understand why the variant of gradient descent does not have what I think. Like normalize the gradient into unit vector

ps. I presume that my Gradient Descent might not work well when the magnitude of gradient is extremely low and could diverge to inf

$\endgroup$

2 Answers 2

1
$\begingroup$

In a gradient descent algorithm, the algorithm proceeds by finding a direction along which you can find the optimal solution. The optimal direction turns out to be the gradient. However, since we are only interested in the direction and not necessarily how far we move along that direction, we are usually not interested in the magnitude of the gradient. A normalized gradient is good enough for our purposes, and we let 𝜂 dictate how far we want to move in the computed direction.

There is no difference between normalized and unnormalized gradient descent (as far as the theory behind the algorithm goes). However, it has a practical impact on the speed of convergence and stability. The choice of one over the other is purely based on the application/objective at hand.

I think this has already been answered here.

$\endgroup$
1
$\begingroup$

The most obvious reason is that a gradient of the norm of 1 is expected to be at learning_rate * 1 away from the loss function minimum. That is averaged of course.

Proper gradient near the minimum approaches to 0 due to cancellation of the first order term in Taylor expansion of Loss function around the minimum. So at least in theory, a step based on a Loss gradient rather than normalised gradient can converge towards the minimum.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.