Adam uses mini batches to optimize. During optimization, you may need go down hill, the cost function, so quickly using a high learning rate. When you reach to points which are near to relatively optimal point you have to reduce the learning rate in order not miss the optimal point. In other words you have to decay learning rate to have more accurate steps by reducing the learning rate. Mini-batch optimizers have multiple steps during one epoch, which all of them may not be true but because they try to minimize the cost for each batch of input data, they finally reach to the relative optimal points.
For each epoch,
TensorFlow uses same learning rate and after finishing epoch, the next epoch will be started using the current learning rate divided by the decay parameter. It should not be negative because you are using gradient descent which implies moving toward low-level places.
Recently I was looking the code of optimizers in
and I found that as the following code:
if self.initial_decay > 0:
lr *= (1. / (1. + self.decay * K.cast(self.iterations,
learning changes each iteration and not each epoch.