4
$\begingroup$

In PyTorch doc, it suggests

torch.optim.lr_scheduler provides several methods to adjust the learning rate based on the number of epochs.

However, from other sources it looks like the learning rate should be adjusted in every optimization step (batch):

So, Should the learning rate in a Learning Rate Scheduler be adjusted at each optimization step (batch) or at each epoch?

Is there a definitive answer to this, or it depends on the model? For transformer models, it looks like the learning rate is adjusted at every step (training batch). In the following example, there are a few thousands of steps so I think it cannot be epochs, is that right? https://nn.labml.ai/optimizers/noam.html enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ Pay attention that number of optimization steps can differ from the number of batches if using gradient accumulation, which is frequently used if you do not have huge computational resources, expecially with bigger model like transformers. $\endgroup$
    – Ciodar
    Jun 19, 2023 at 17:11

1 Answer 1

2
$\begingroup$

It very much depends on your training setup. More specifically, the interaction between learning rate and batch-size plays an important role on how the learning rate decay affects the learning dynamics.

In a typical deep learning setup, you will optimise the mean error. If you plot the loss curves for different batch-sizes (both in terms of number of updates and number of epochs), you should get a figure like the one below:

learning curves when training on mean error

On one side, we see that the batch-size has practically no effect on the learning dynamics in terms of number of updates. On the other side, the changes in batch-size have a noticeable effect on the dynamics on the learning dynamics in terms of number of epochs. Therefore, if you specify the decay in terms of number of updates, the decay should kick in at (roughly) the same point in the learning process no matter what batch-size you choose (i.e. the decay is independent of the batch-size). When specifying the decay in terms of epochs, on the other hand, you would probably need to update the decay parameters when changing the batch-size to get similar effects.

Of course, it is also possible to optimise the sum of errors. Plotting the average validation loss curves for different batch-sizes (similar as before), you get the following figure:

learning curves when training on sum of errors

Now, the different batch size has an effect on the learning dynamics in terms of the number of updates. Therefore, you would need to specify the decay in terms of the number of epochs if you wish to be batch-size independent.

Obviously, this is just one side of the story and there might be other interactions. However, if you want your learning rate decay to be independent from your batch-size, you should specify it in terms of updates (epochs) if you are optimising the average (sum) over errors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.