7
$\begingroup$

I'm currently training a CNN with Keras and I'm using the Adam optimizer. My plan is to gradually reduce the learning rate after each epoch. That's what I thought the decay parameter was for. For me, the documentation does not clearly explain how it works:

decay: float >= 0. Learning rate decay over each update.

However, when looking at the used learning rate in tensorboard, it stays the same as the initial learning rate. So, how does this decay parameter actually work?

$\endgroup$
0

1 Answer 1

3
$\begingroup$

Adam uses mini batches to optimize. During optimization, you may need go down hill, the cost function, so quickly using a high learning rate. When you reach to points which are near to relatively optimal point you have to reduce the learning rate in order not miss the optimal point. In other words you have to decay learning rate to have more accurate steps by reducing the learning rate. Mini-batch optimizers have multiple steps during one epoch, which all of them may not be true but because they try to minimize the cost for each batch of input data, they finally reach to the relative optimal points.

For each epoch, TensorFlow uses same learning rate and after finishing epoch, the next epoch will be started using the current learning rate divided by the decay parameter. It should not be negative because you are using gradient descent which implies moving toward low-level places.


Recently I was looking the code of optimizers in Keras and I found that as the following code:

if self.initial_decay > 0:
       lr *= (1. / (1. + self.decay * K.cast(self.iterations,
                                             K.dtype(self.decay))))

learning changes each iteration and not each epoch.

$\endgroup$
5
  • $\begingroup$ I'm not sure which information I can filter from your post. I'm aware of the first thing you are mentioning. But this doesn't help me with the actual problem. The second aspect is what I plan to do but obviously this is not happening with my current implementation of the adam optimizer. I set the lr parameter to 0.003 and the decay parameter to 0.0002 . However, according to the tensorboard graphs it keeps staying at 0.003. Neither a deduction of the decay rate nor a division by it. $\endgroup$
    – TheDude
    Commented Dec 29, 2017 at 16:30
  • $\begingroup$ are you sure about setting the decay parameter > 1? If that's the case, then the documentations is really messed up which I cannot imagine actually. And if the lr is truly divided by the decay rate, then my loss should be exploding with my current settings at some point, right? Although I'm Tensorflow as backend, I'm still on keras if that matters. Just to make that clear. But I will try. My epochs is currently set to 50 but I'm still experimenting. I have just the feeling that my progress at the beginning has to be more aggressive. $\endgroup$
    – TheDude
    Commented Dec 29, 2017 at 17:01
  • $\begingroup$ Using a decay parameter of 0.0002 is quiet small and the decay will be minimal. it is not gonna explode it will decay but the decay amount is negligible. new learning rate = old learning rate / (1 + decay * iterations)) for the first iteration new learning rate = 0.003 * 1/(1+0.0002) = 0.0029994 BTW, the initial learning rate is also small (0.003) which will result in a slower learning. $\endgroup$
    – ahmed
    Commented Jun 13, 2018 at 18:11
  • 1
    $\begingroup$ @ahmed Actually the learning rate setup highly depends on the task. $\endgroup$ Commented Jun 17, 2018 at 13:11
  • 1
    $\begingroup$ @Media OK. Now I see how come my model can't be properly trained by adding decay into the Adam optimizer. $\endgroup$
    – pitfall
    Commented Aug 20, 2018 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.