In PyTorch doc, it suggests
torch.optim.lr_schedulerprovides several methods to adjust the learning rate based on the number of epochs.
However, from other sources it looks like the learning rate should be adjusted in every optimization step (batch):
- Warmup steps in deep learning
So, Should the learning rate in a Learning Rate Scheduler be adjusted at each optimization step (batch) or at each epoch?
Is there a definitive answer to this, or it depends on the model? For transformer models, it looks like the learning rate is adjusted at every step (training batch). In the following example, there are a few thousands of steps so I think it cannot be epochs, is that right? https://nn.labml.ai/optimizers/noam.html