1
$\begingroup$

I've seen the following happen in our training: (numbers are paraphrased)

  • epoch 10 - val loss 0.0500 - loss 0.0450
  • epoch 11 - val loss 0.0400 - loss 0.0400
  • epoch 12 - val loss 0.0420 - loss 0.0410

Now clearly epoch 11 is better than epoch 12. I assumed epoch 13 would start with the weights of epoch 11, but have been told they do not.

In my opinion worst of all, our training pipeline is set up with ReduceLROnPlateau lowering the learning weight after 4 epochs of no improvement. Now say epoch 15 has a validation loss of 0.0415 and a loss of 0.0405, we could've better trained on with the weights of epoch 11.

Why are new epochs not started with the weights of the previous best epoch?

$\endgroup$

2 Answers 2

3
$\begingroup$

A new epoch starts with the same weights as the previous epoch but with different mini-batch of data. Model performance will have small oscillations across mini-batches for that reason.

Always starting each epoch with the best overall weights might result in staying in a local optimum. Gradient descent is already prone to staying in a local optimum, at the expense of finding a better optimum. Encouraging continued exploration of weights is one of the goals of training.

There is always a chance that both training and validation loss will do down. It is okay that training performance gets worse, there is no need for monotonic improvement in training.

At the end of the training, you should select the weights that have the lowest validation loss.

$\endgroup$
1
$\begingroup$

Because you would probably get a similar result. Remember that the last result (valid loss of 0.0420) also started from the point where you got this better result (valid loss of 0.0400), and ended up being worse. A random restart from the "good checkpoint" would probably give you the same situation.

It seems you have here an overfitting problem. There are techniques to address this kind of problem, like regularization or early stopping.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.