1
$\begingroup$

The Adam optimizer has four main hyperparameters. For example, looking at the Keras interface, we have:

keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)

The first hyperparameter is called step size or learning rate. In theory, an adaptive optimization method should automatically modify the learning rate during optimization. Thus, I would expect lr to be a not very influential hyperparameter, which I can remove from the list of hyperparameters I have to tune on the validation set, thus saving some time.

Does this hold true in practice? I.e., is it true that, at least for a class of problems (say, image recognition), lr doesn't affect the optimization a lot, and thus we can just leave it at the default value of 0.001? Or is it still as extremely influential, as it is for SGD with momentum?

EDIT: to clear a misunderstanding brought up in an answer, learning rate and step size are synonymous. See the algorithm definition in the Adam paper: $\alpha$, whose default value is 0.001, is clearly named the step size. I reckon it's probably not a great name (the actual size of the step in the parameter space depends on the accumulated first order and second order momentums, as well as on the gradient, of course) but unfortunately such misleading terminology is the norm in optimization (at least in Deep Learning papers).

enter image description here

$\endgroup$
2
$\begingroup$

the lr (learning rate) parameter is the MOST influencial parameter for Adam (or any other optimization algorithm). I It does not mean step size... as that is not controlled by Adam but by the neural network and how you feed batches into it.

So yes, lr is very, very important, and it is the learning rate, not the step size.

UPDATE

Actually as mentioned by OP, the learning rate and step size are the same thing (my bad, I confused it with the batch size).

Generally, my recommendation is to try with learning rates of 0.0001, 0.001, 0.01 and 0.1.

The rule of thumb is that a smaller learning rate provides a much stable learning curve... but a much slower one while a larger learning rate will converge quicker... but it will become unstable. So you will have to do a bit of exploration. I normally use 0.001 and 0.0001 but that depends a lot on the problem you are dealing with.

Also, as compared with SGD, Adam includes momentum, essentially taking into account past behaviour (the further away such behaviour is in the past, the least relevant it is) that is controlled by the beta params, but I do not recommend to modify them, as normally it is not required. So, although Adam is a more stable learning method (and normally a quicker one) than SGD, the learning rate it is still the most fundamental parameter to play with.

$\endgroup$
3
  • $\begingroup$ it absolutely is the step size. See the original paper, and the description in my edited question. $\endgroup$ – DeltaIV Dec 30 '18 at 23:20
  • 1
    $\begingroup$ You are absolutely right, I updated the question. $\endgroup$ – Juan Antonio Gomez Moriano Dec 31 '18 at 2:59
  • $\begingroup$ perfect, thanks. Sure, it definitely depends on the problem. I was hoping that for Adam it wouldn't be necessary to play with lr, but your answer confirms me that it is indeed necessary. Thanks! And an Happy New Year :-) $\endgroup$ – DeltaIV Dec 31 '18 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.