4
$\begingroup$

I'm currently training a CNN with Keras and I'm using the Adam optimizer. My plan is to gradually reduce the learning rate after each epoch. That's what I thought the decay parameter was for. For me, the documentation does not clearly explain how it works:

decay: float >= 0. Learning rate decay over each update.

However, when looking at the used learning rate in tensorboard, it stays the same as the initial learning rate. So, how does this decay parameter actually work?

$\endgroup$
3
$\begingroup$

Adam uses mini batches to optimize. During optimization, you may need go down hill, the cost function, so quickly using a high learning rate. When you reach to points which are near to relatively optimal point you have to reduce the learning rate in order not miss the optimal point. In other words you have to decay learning rate to have more accurate steps by reducing the learning rate. Mini-batch optimizers have multiple steps during one epoch, which all of them may not be true but because they try to minimize the cost for each batch of input data, they finally reach to the relative optimal points.

For each epoch, TensorFlow uses same learning rate and after finishing epoch, the next epoch will be started using the current learning rate divided by the decay parameter. It should not be negative because you are using gradient descent which implies moving toward low-level places.


Recently I was looking the code of optimizers in Keras and I found that as the following code:

if self.initial_decay > 0:
       lr *= (1. / (1. + self.decay * K.cast(self.iterations,
                                             K.dtype(self.decay))))

learning changes each iteration and not each epoch.

$\endgroup$
  • $\begingroup$ I'm not sure which information I can filter from your post. I'm aware of the first thing you are mentioning. But this doesn't help me with the actual problem. The second aspect is what I plan to do but obviously this is not happening with my current implementation of the adam optimizer. I set the lr parameter to 0.003 and the decay parameter to 0.0002 . However, according to the tensorboard graphs it keeps staying at 0.003. Neither a deduction of the decay rate nor a division by it. $\endgroup$ – TheDude Dec 29 '17 at 16:30
  • $\begingroup$ are you sure about setting the decay parameter > 1? If that's the case, then the documentations is really messed up which I cannot imagine actually. And if the lr is truly divided by the decay rate, then my loss should be exploding with my current settings at some point, right? Although I'm Tensorflow as backend, I'm still on keras if that matters. Just to make that clear. But I will try. My epochs is currently set to 50 but I'm still experimenting. I have just the feeling that my progress at the beginning has to be more aggressive. $\endgroup$ – TheDude Dec 29 '17 at 17:01
  • $\begingroup$ Using a decay parameter of 0.0002 is quiet small and the decay will be minimal. it is not gonna explode it will decay but the decay amount is negligible. new learning rate = old learning rate / (1 + decay * iterations)) for the first iteration new learning rate = 0.003 * 1/(1+0.0002) = 0.0029994 BTW, the initial learning rate is also small (0.003) which will result in a slower learning. $\endgroup$ – ahmed Jun 13 '18 at 18:11
  • 1
    $\begingroup$ @ahmed Actually the learning rate setup highly depends on the task. $\endgroup$ – Media Jun 17 '18 at 13:11
  • 1
    $\begingroup$ @Media OK. Now I see how come my model can't be properly trained by adding decay into the Adam optimizer. $\endgroup$ – pitfall Aug 20 '18 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.