5
$\begingroup$

The formula for Gradient Descent is as follows: $$ \mathbf{w} := \mathbf{w} - \alpha\; \triangledown C $$

The gradient itself points in the direction of steepest ascent, therefore it is logical to go in the opposite direction by subtracting it. But besides the direction, the gradient also has a magnitude which actually doesn't say anything about the path to the optimum.

My question is, is this considered a weakness of gradient descent and the reason for the learning rate?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

The magnitude doesn't say anything about the path to the optimum, in fact nothing in gradient decent knows anything about the optimum and the algorithm relays on local information only. However if you assume that the space of available solutions is relatively smooth and doesn't contain sharp drops, you can treat the magnitude of the gradient as a score of your confidence in your current surrounding.

So if the gradient magnitude is high, you are fairly certain that you are far from a good solution and you can make big steps and if the magnitude is small, than you are closer to a good solution and should make smaller changes.

The learning rate is just a hyper-parameter that controls how much we are adjusting the weights of our network with respect that said loss gradient. It can also be viewed in terms of your confidence in the surrounding solutions space. It also helps the model to converge deeper into a specific minimum point (as we reduce the learning rate during training).

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the great answer. I have two questions regarding it. 1) Are Cost functions known to have smooth functions? 2) What do you mean by, the learning rate can be viewed in terms of confidence. Do you mean that if we set the lr to say: 0.1 we hereby express that we are highly confident? Would that only be a confidence we as a person have? $\endgroup$
    – oezguensi
    Dec 26, 2018 at 15:12
  • 1
    $\begingroup$ 1) Don't confuse my use of the term smooth with official (which is continuity of the derivatives). I only meant to say that usually there are no huge drops and elevations in the solutions space, i.e. a small change to the weights usually results in a small change in the cost. 2) It is a relative term, so for the same model, if you use a big learning rate, it means that you think that you are far from a good solution and you want to make big changes. Small learning rate means that you are confident you are close to a solution and you want to make small changes to get as close as you can to it. $\endgroup$
    – Mark.F
    Dec 26, 2018 at 15:50
  • $\begingroup$ Ok, thanks for the clarification. I will read more about "continuity of derivatives". $\endgroup$
    – oezguensi
    Dec 26, 2018 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.