Take the 2-minute tour ×
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It's 100% free, no registration required.

I'm currently working on implementing Stochastic Gradient Descent (SGD) for neural nets using backpropagation, and while I understand its purpose I have some questions about how to choose values for the learning rate.

  • Is the learning rate related to the shape of the error gradient, as it dictates the rate of descent?
  • If so, how do you use this information to inform your decision about a value?
  • If it's not what sort of values should I choose, and how should I choose them?
  • It seems like you would want small values to avoid overshooting, but how do you choose one such that you don't get stuck in local minima or take to long to descend?
  • Does it make sense to have a constant learning rate, or should I use some metric to alter its value as I get nearer a minimum in the gradient?

In short: How do I choose the learning rate for SGD?

share|improve this question

2 Answers 2

up vote 14 down vote accepted
  • Is the learning rate related to the shape of the error gradient, as it dictates the rate of descent?

    • In plain SGD, no. A global learning rate is used which is indifferent to the error gradient. However the intuition you are getting at has inspired various modifications of the SGD update rule.
  • If so, how do you use this information to inform your decision about a value?

    • Adagrad is the most widely known of these and scales a global learning rate η on each dimension based on l2 norm of the history of the error gradient gt on each dimension:

      enter image description here

    • Adadelta is another such training algorithm which uses both the error gradient history like adagrad and the weight update history and has the advantage of not having to set a learning rate at all.

  • If it's not what sort of values should I choose, and how should I choose them?

    • Setting learning rates for plain SGD in neural nets is usually a process of starting with a sane value such as 0.01 and then doing cross validation to find an optimal value. Typical values range over a few orders of magnitude from 0.0001 up to 1.
  • It seems like you would want small values to avoid overshooting, but how do you choose one such that you don't get stuck in local minima or take to long to descend? Does it make sense to have a constant learning rate, or should I use some metric to alter its value as I get nearer a minimum in the gradient?

    • Usually the value that's best is near the highest stable learning rate and learning rate decay/annealing (either linear or exponentially) is used over the course of training. The reason behind this is that early on there is a clear learning signal so aggressive updates encourage exploration while later on the smaller learning rates allow for more delicate exploitation of local error surface.
share|improve this answer
    
In practice, you will use a learning rate with adadelta. On some problems it does not work without. –  bayer Jun 29 at 18:18

Below is a very good note (page 12) on learning rate in Neural Nets (Back Propagation) by Andrew Ng. You will find details relating to learning rate.

http://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf

For your 4th point, you're right that normally one has to choose a "balanced" learning rate, that should neither overshoot nor converge too slowly. One can plot the learning rate w.r.t. the descent of the cost function to diagnose/fine tune. In practice, Andrew normally uses the L-BFGS algorithm (mentioned in page 12) to get a "good enough" learning rate.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.