Given the number of epochs, batch size and learning rate, is there a formula by which I can calculate the learning rate decay in mini batch SGD?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
even though there are some learning rate decay schemes for Standard Gradient descent, convergence of these methods are not justified beyond convex functions or functions whose gradients are globally Lipschitz. Even for those latter maps, when you do not have a good lower bound for the Lipschitz constant L, basically you have no way to say which learning rate is good.
In my joint paper, we proposed a method to modify learning rate using the backtracking version of Gradient Descent, which works quite stably and effectively. More details about the theoretical background and the accompanying source codes can be found in my answer in this link:
