# Why a sign of gradient (plus or minus) is not enough for finding a steepest ascend?

Consider a simple 1-D function $$y = x^2$$ to find a maximum with the gradient ascent method.

If we start in point 3 on x-axis: $$\frac{\partial f}{\partial x} \biggr\rvert_{x=3} = 2x \biggr\rvert_{x=3} = 6$$

This means that a direction in which we should move is a $$6$$.

Gradient ascent gives rule to update: x = old_x + learning_rate * gradient

What I can't understand why we need to multiply a learing_rate with gradient. Why we can't just use x = old_x + learning_rate * sign(gradient).

Because if we made a learning_rate step in a positive direction it is already a maximum switch of x we can make.

I know the reasoning behind finding maximum direction in this equation:

$$grad(π(π))β π£β=|grad(π(π))||π£β|cos(π)$$

But I can't undestand why just to accept a sign of gradient (plus or minus) is not enough for ascending.

Using only sign of gradient is a way to go, but might result in slow convergence. Nevertheless it is a valid variation of the method.

Sign-based optimization methods have become popular in machine learning due to their favorable communication cost in distributed optimization and their surprisingly good performance in neural network training. Furthermore, they are closely connected to so-called adaptive gradient methods like Adam. Recent works on signSGD have used a non-standard "separable smoothness" assumption, whereas some older works study sign gradient descent as steepest descent with respect to the ββ-norm. In this work, we unify these existing results by showing a close connection between separable smoothness and ββ-smoothness and argue that the latter is the weaker and more natural assumption. We then proceed to study the smoothness constant with respect to the ββ-norm and thereby isolate geometric properties of the objective function which affect the performance of sign-based methods. In short, we find sign-based methods to be preferable over gradient descent if (i) the Hessian is to some degree concentrated on its diagonal, and (ii) its maximal eigenvalue is much larger than the average eigenvalue. Both properties are common in deep networks.

• thank you, but question is about gradient, not learning rate, I understand the necessity of learning_rate. Jan 16, 2021 at 16:48
Think about the convergence behavior of your algorithm (assuming a fixed learning rate) β once it gets x to within learning_rate of the optimum, it will jump to the other side, and the sign will change. Then it will jump back to exactly the previous value, and oscillate between those two values until you choose to terminate it, never getting any closer to the true optimum.