I am training a transformer-based model using Pytorch. The training loss decreases until it hits a floor, which is expected. However, the validation loss decreases to a minimum then starts increasing.
Furthermore, when changing the hyperparameters and features used during training (shown in the chart below), we can see that despite having different minimum validation loss, the validation loss eventually increases to a similar value given enough epochs.
Why do all the validation losses end up with similar values after sufficient epochs has passed?
Can switching optimizers in the middle of training help? Such as starting off with Adam then switching to SGD with momentum? Or adding warmups phases? Or learning rates schedules (and does Adam require LR schedules?)?