From my background, I understand that the purpose of having a learning rate (α) is to normalize the magnitude of gradient (▽J), so the step size can properly converge the local minima
Since α is arbitrary, so we have to do find the best hyperparameter learning rate (α).
Perhap, we do not have to hyper-parameter tuning when we normalize the gradient into a unit vector in which its magnitude always 1
The traditional gradient descent:
My gradient descent:
Again, back to my problem, I do not understand why the variant of gradient descent does not have what I think. Like normalize the gradient into unit vector
ps. I presume that my Gradient Descent might not work well when the magnitude of gradient is extremely low and could diverge to inf