How would one go about selecting the validation set used to evaluate trained models by an automated system, in order to ensure that each new model is at least as good as, or better than the previous model?
Lets say that in this case we have a regression problem, and we can depend on a single performance metric such as RMSE.
I'm working on a program that automatically trains, tests and if the tests passed, deploys a new deep learning model every x new data samples/time.
Each training is done on the original dataset (labeled data) and the new dataset which is labeled by predictions of the previous model.
These were the options I came up with and some pros/cons.
Option A:
Create a validation set once, always apply this set
+ Always have a consistent benchmark
- Validation set might get 'out of date' in cases where the relevancy of data might change with time
Option B:
Create an initial validation set, add X percentage of new training samples to the original set
+ Consistent time spread of samples
Option C:
Create a random validation set every training
+ Benchmark is most likely to contain a relevant set of samples (That are in data)
- Test results can vary each training session
Option D:
Cross-validation
+ Benchmark is most likely to contain a relevant set of samples (That are in data)
+ Likely least variance between each training session
- Computationally expensive, even more so for deep learning models
