I am training a transformer-based model using Pytorch. The training loss decreases until it hits a floor, which is expected. However, the validation loss decreases to a minimum then starts increasing.

Furthermore, when changing the hyperparameters and features used during training (shown in the chart below), we can see that despite having different minimum validation loss, the validation loss eventually increases to a similar value given enough epochs.

  1. Why do all the validation losses end up with similar values after sufficient epochs has passed?

  2. Can switching optimizers in the middle of training help? Such as starting off with Adam then switching to SGD with momentum? Or adding warmups phases? Or learning rates schedules (and does Adam require LR schedules?)?

enter image description here enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.