Gradient descent method

Question

If we suppose that this is formula for gradient descent method:

$$x_{n+1}=x_n-\lambda\cdot{{df(x)}\over{dx}},\ n=0,1,2,3,...$$

Since there is no exact value that we subtract instead of derivative, does it mean that we subtract value of derivative and use it only for controlling direction of next position of x? And why do we subtract derivative but not any other value that is depends on x?

Answer 1

Suppose I want to find the minimum of a function $f(x)$ around $x_m$. Then I have 3 options:

1. minimum is $x_m$ within some accuracy (end of procedure)
2. minimum is to the right of $x_m$ (and update my initial guess appropriately)
3. minimum is to the left of $x_m$ (and update my initial guess appropriately)

So:

1. If $x_m$ is close (within desired accuracy) to the minimum then derivative at this point will be (approximately) zero (basic analysis).
2. If minimum is to the right of $x_m$ (lets say $x_{m'}$) then $f'(x_m)$ will have negative slope (negative of derivative points to the direction of minimum).
3. If minimum is to the left of $x_m$ (lets say $x_{m'}$) then $f'(x_m)$ will have positive slope (negative of derivative points to the direction of minimum).

So far derivative seems a good choice to use to update my initial guess. If magnitude is also taken into account we have:

1. If $x_m$ is close (within desired accuracy) to the minimum then derivative at this point will be (approximately) zero.
2. If minimum is to the right or left of $x_m$ (lets say $x_{m'}$) then $f'(x_m)$ will have non-zero magnitude that becomes greater as the $x_m$ is furthest from minimum.

So derivative magnitude can also be used.

So far we have observed and deduced that: $x_{m'}-x_m = \Delta x \sim -f'(x_m)$ or that: $x_{m'} = x_m - \lambda f'(x_m)$ is a good scheme to update my guess.

$\lambda$ (learning rate) is a necessary parameter that satisfies two things:

1. if $x$ and $f'(x)$ have different physical dimensions then it scales the derivative appropriately to match the dimensions of $x$.
2. It allows to adjust the learning rate so as not to overshoot the desired minimum or go too slow, in case derivative has some critical points.

References:

1. Gradient descent
2. Why sign of gradient is not enough for finding steepest ascent