Should I use validation data and val_loss when training final model?

Question

I am training a keras model that utilizes early_stopping in order to prevent overfitting. This requires that I set aside a validation dataset.

My task requires that I keep my training and validation split by time, so that all samples in my validation set occur after the point in time of those in my training set.

My challenge is that the examples in my validation (by definition the most recent examples in time) are very important for my prediction task and I would like to use them to train a final model. From all I can see, it seems that in general it is recommended to train a final model (to be released to production) on all data available, after model configuration has been decided upon in a traditional train/test period (see here).

However, if I use all of my data to train a final model, I no longer can utilize early_stopping, since I will not have any validation set (it will be being used for training).

I could randomly sample a subset of my training data to use for validation (instead of using the most recent data as I was during training/testing), but then I worry that due to the time series dynamic of the problem I am running the risk of data leakage.

My question really boils down to:

What is the preferred way to train a final, production model when in Keras (or another framework)?

Thanks!

Answer 1

• Especially for time series work, yes, use your full dataset for training your final model.
• Keep your number of epochs the same as the best performance on your val_loss.
• If you want, you can remove the same period of time from the start of the training data, to ensure the model is given a consistent number of samples to learn over.

This is a big challenge when shipping models to production, as you now have no validation set, how do you know how well it is performing?

• This is where you need to get creative, and use different time-series k-fold splitting strategies.

When working on TS problems that you want to ship to production, I like to use a training, validation and holdout set. So you can test your models true performance. Otherwise you will be over-fitting on your validation set, through early stopping. (You are leaking information from your validation set, into the keras model training process)

Hope this helps.