Why do we move in the negative direction of the gradient in Gradient Descent?

Question

It is said that backpropagation, with Gradient Descent, seeks to minimize a cost function using the formula:

$$ W_{new} = W_{old} - learningRate \cdot \frac{\partial E}{\partial W} $$

My question is, if the derivate indicates in which direction the function (the graph of the error with respect to the weights) is decreasing, then why subtract from an already negative gradient?

Why not allow the current direction of the gradient (negative lets say) to be the driving factor for updating the weights:

$$ W_{new} = W_{old} + learningRate \cdot (-gradient) $$

Answer 1

Consider a simple example where the cost function to be a parabola $y=x^2$ which is convex(ideal case) with a one global minima at $x=0$

Here your $y$ is the independent variable and $x$ is the dependent variable, analogus to the weights of model that you are trying to learn.

This is how it would look like.

Let's apply gradient descent to this particular cost function(parabola) to find it's minima.

From calculus it is clear that $dy/dx = 2*x$. So that means that the gradients are positive in the $1^{st}$ quadrant and negative in the $2^{nd}$. That means for every positive small step in x that we take, we move away from origin in the $1^{st}$ quadrant and move towards the origin in the $2^{nd}$ quadrant(step is still positive).

In the update rule of gradient descent the '-' negative sign basically negates the gradient and hence always moves towards the local minima.

• $1^{st}$ quadrant -> gradient is positive, but if you use this as it is you move away from origin or minima. So, the negative sign helps here.
• $2^{nd}$ quadrant -> gradient is negative, but if you use this as it is you move away from origin or minima(addition of two negative values). So, the negative sign helps here.

Here is a small python code to make things clearer-

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-4, 4, 200)
y = x**2

plt.xlabel('x')
plt.ylabel('y = x^2')
plt.plot(x, y)

# learning rate
lr = 0.1
np.random.seed(20)
x_start = np.random.normal(0, 2, 1)
dy_dx_old = 2 * x_start
dy_dx_new = 0

tolerance = 1e-2
# stop once the value has converged
while abs(dy_dx_new - dy_dx_old) > tolerance:
    dy_dx_old = dy_dx_new
    x_start = x_start - lr * dy_dx_old
    dy_dx_new = 2 * x_start
    plt.scatter(x_start, x_start**2)
    plt.pause(0.5)
plt.show()

Answer 2

Let $F : \mathbb{R}^{n} \rightarrow \mathbb{R}$ be a continuous differentiable function and $d \in \mathbb{R}^{n}$. Then $d$ is called a descent direction at position $p \in \mathbb{R}^{n}$, if there is a $R > 0 $ such that $F(p+rd) < F(p)$ for all $r \in (0,R)$.

In simple terms: If we move $p$ in direction of $d$ we can reduce the value of $F$.

Now $d$ is a descent direction at $p$, if $\nabla F(p)^T rd < 0 $ since by definition of gradient $F(p+rd) - F(p) = \nabla F(p)^Trd$ and in order to reduce the function value, we need to have $\nabla F(p)^T rd < 0 $:

For $f(r):= F(p+rd)$ we have $f'(t) = \nabla F(p+rd)^T d$. By assumption, $f'(0) < 0$ holds.

Since $f'(0) = \lim_{h \rightarrow 0} \frac{f(h)-f(0)}{h}$, we conclude that $d$ must be descent direction.

Therefore, setting $d := -\nabla F(p)$, we have $\nabla F(p)^T (-\nabla F(p)) = - ||\nabla F(p)||_{2}^{2} < 0 $, if $p$ is not a stationary point.

In particular, we can choose an $p' = p + r'd$ with $F(p') < F(p)$. This shows that using the negative gradient makes sense.

Answer 3

Computing the gradient gives you the direction where the function increases the most. Consider f: x -> x^2 , the gradient in x=1 is 2, if you want to minimize the function you need to go in the direction of -2, same with x=-1 as the gradient is -2.

And as gradients are usually vectors, I don't know what a positive or negative gradient would be if gradient is something like (-1, 1).

https://builtin.com/data-science/gradient-descent