If we suppose that this is formula for gradient descent method:

$$x_{n+1}=x_n-\lambda\cdot{{df(x)}\over{dx}},\ n=0,1,2,3,...$$

Since there is no exact value that we subtract instead of derivative, does it mean that we subtract value of derivative and use it only for controlling direction of next position of x? And why do we subtract derivative but not any other value that is depends on x?

• does this answer your question? and if not then where is the problem? Jun 11, 2021 at 14:04
• @NikosM., The problem is that I'm confused with, is the way we choose derivative as a thing to subtract Jun 11, 2021 at 14:09
• The wikipedia article explains exactly this very well. It is mathematicaly sound as well. Indeed the derivative value and direction points to the direction of descent. So it all makes sense right. Note that there are other methods which are not linear but still use gradient (eg newton-raphson method) or not use gradient at all (eg stochastic optimisation) Jun 11, 2021 at 14:12

Suppose I want to find the minimum of a function $$f(x)$$ around $$x_m$$. Then I have 3 options:

1. minimum is $$x_m$$ within some accuracy (end of procedure)
2. minimum is to the right of $$x_m$$ (and update my initial guess appropriately)
3. minimum is to the left of $$x_m$$ (and update my initial guess appropriately)

So:

1. If $$x_m$$ is close (within desired accuracy) to the minimum then derivative at this point will be (approximately) zero (basic analysis).
2. If minimum is to the right of $$x_m$$ (lets say $$x_{m'}$$) then $$f'(x_m)$$ will have negative slope (negative of derivative points to the direction of minimum).
3. If minimum is to the left of $$x_m$$ (lets say $$x_{m'}$$) then $$f'(x_m)$$ will have positive slope (negative of derivative points to the direction of minimum).

So far derivative seems a good choice to use to update my initial guess. If magnitude is also taken into account we have:

1. If $$x_m$$ is close (within desired accuracy) to the minimum then derivative at this point will be (approximately) zero.
2. If minimum is to the right or left of $$x_m$$ (lets say $$x_{m'}$$) then $$f'(x_m)$$ will have non-zero magnitude that becomes greater as the $$x_m$$ is furthest from minimum.

So derivative magnitude can also be used.

So far we have observed and deduced that: $$x_{m'}-x_m = \Delta x \sim -f'(x_m)$$ or that: $$x_{m'} = x_m - \lambda f'(x_m)$$ is a good scheme to update my guess.

$$\lambda$$ (learning rate) is a necessary parameter that satisfies two things:

1. if $$x$$ and $$f'(x)$$ have different physical dimensions then it scales the derivative appropriately to match the dimensions of $$x$$.
2. It allows to adjust the learning rate so as not to overshoot the desired minimum or go too slow, in case derivative has some critical points.

References:

• "If minimum is to the right or left of 𝑥𝑚 (lets say 𝑥𝑚′) then 𝑓′(𝑥𝑚) will have non-zero magnitude that becomes greater as the 𝑥𝑚 is furthest from minimum." And same if 𝑥𝑚 is closer from minimum, then magnitude becomes lower, right? Jun 11, 2021 at 15:26
• Right in general, except extreme cases Jun 11, 2021 at 15:28
• Thanks for your answer, that was the problem! Jun 11, 2021 at 15:30
• I am glad it helped. Jun 11, 2021 at 15:31