1
$\begingroup$

I've been experimenting with reinforcement learning and using the train_on_batch method of tf.keras.models.Model to update the models. At this point I've done a fair amount of reading on RL but haven't seen anything on learning rate decay specific to RL. Trying to read a little more about learning rate decay and Adam makes me think that I probably don't fully understand how various optimizers operate over batches in Tensorflow.

Taking a step back from RL, it's pretty evident that the effective learning rate decreases over the batches in each epoch with a vanilla deep learning model. That is, towards the beginning of an epoch, the loss jumps more significantly and then stabilizes towards the end of the epoch. This can be seen whether using Adam, SGD, etc.

Having read about learning rate decay when I first started with deep learning and after seeing this stabilizing phenomenon, I assumed that Tensorflow was decaying the learning rate automatically under the hood, however, that doesn't seem to be the case based on the answers here:

https://stackoverflow.com/questions/33919948/how-to-set-adaptive-learning-rate-for-gradientdescentoptimizer.

Looking at SGD specifically, I don't see where the default SGD settings allow for any kind of decay, but I see the same stabilization towards the end of an epoch with SGD as with Adam (Adam does have decay parameters). Here are the SGD defaults:

tf.keras.optimizers.SGD(
    learning_rate=0.01, momentum=0.0, nesterov=False, name='SGD', **kwargs
)

So my questions are:

  1. What's driving this stabilization towards the end of epochs in vanilla deep learning models? What is Tensorflow doing under the hood?
  2. How can I replicate this when using the train_on_batch method for reinforcement learning with Adam?

To clarify on #2, I think manually decaying SGD would be pretty straightforward, but because Adam has additional parameters, replicating whatever decay is happening over batches with model.fit might be more complicated, depending on the answer to #1.

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.