I am not an expert and do not have theoretical justification for that, but it seems to me that the smaller network error is, the smaller learning rate should be.

Is there an algorithm to dynamically update learning rate based on total error of the network without relying on any hyper-parameters ?


Your intuition is on point, and shrinking the learning rate like this is often referred to as "annealing". But linking the learning rate to error magnitude neglects certain problematic error surface topologies. An excellent motivating example is the Rosenbrock "Banana" Function, which is often used as a test case for optimization algorithms. The "banana" is a low error valley which hides the global minimum. If an optimization path finds its way into this valley, the path to the global minimum is along a nearly flat gradient.

The Rosenbrock Function

If you use an optimization algorithm that naively shrinks the learning rate relative to the error, you're going to get stuck as soon as you hit the valley. On the one hand: congrats! You've achieved a low error solution. But you're not necessarily anywhere near the global minimum. So how can we do better?

An approach used by modern gradient-based methods like Adagrad, RMSProp, and Adam is to separately assign learning rates to each parameter, and tie the learning rate to the magnitude of the respective parameter's update. The Stanford CS231n lecture notes explains:

Adagrad is an adaptive learning rate method originally proposed by Duchi et al..

# Assume the gradient dx and parameter vector x
cache += dx**2
x += - learning_rate * dx / (np.sqrt(cache) + eps)

Notice that the variable cache has size equal to the size of the gradient, and keeps track of per-parameter sum of squared gradients. This is then used to normalize the parameter update step, element-wise. Notice that the weights that receive high gradients will have their effective learning rate reduced, while weights that receive small or infrequent updates will have their effective learning rate increased.

  • $\begingroup$ shall I add eps to denominator only if np.sqrt(cache) equals 0 ? $\endgroup$
    – koryakinp
    Mar 8 '18 at 19:39
  • $\begingroup$ No, you always add eps to the denominator in adagrad. If you want to learn more about that formula, just follow the link I gave you. $\endgroup$
    – David Marx
    Mar 8 '18 at 21:31
  • $\begingroup$ Ok, thank you sir $\endgroup$
    – koryakinp
    Mar 8 '18 at 21:52
  • $\begingroup$ Also do search for Cosine Annealing for Learning Rates $\endgroup$
    – Aditya
    Mar 10 '18 at 0:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.