When to stop the final model training?

Question

Let's say I'm participating in a Kaggle image recognition competition.

Firstly, I create a train/validation split and find the good hyperparameters for my model. Here the stopping criterion is when the validation loss stops decreasing and starts increasing, something like EarlyStopping Keras callback.

When I have found decent hyperparameters, I want to train my model on the full dataset because I want the best performance for the competition. But what is the stopping criterion for the training now? I don't have a validation loss anymore to track model overfitting.

One strategy I can think of is to record the epoch number when my previous model starting overfitting and train the final model to this number of epochs. For example, during training with the validation subset my model started overfitting after the 15th epoch, so I am going to train my final model on 15 epochs.

Is this strategy good? What other stopping criteria can I use for the final model training?

Answer 1

I've been thinking along these lines - but I don't believe the epoch count is fixed in the way you suggest - so I tried the approach of training on a 70-30 split, an 80-20 split and a 90-10 split and extrapolating the epoch count to a 100-0 split. This seems to work - but it is a lot of throw away training which might be fine to get a good kaggle result, but is very wasteful otherwise - particularly if you change your sampling to get more than one result at each train/validate split pair.