When training a final model for production, it's often recommended to train on all available data (train + dev + test), as discussed here.
I'm training a deep learning model. I typically save and use the best-performing model checkpoint based on performance on a held-out data set. During hyperparameter tuning, I used the dev set performance to pick the best checkpoint in a given experiment. I also have an untouched test set.
For the final model, one option would be to train on the combined training and dev sets, and use the test set to choose the best checkpoint. But what I'd really like to do is train on ALL the data. However, without a held-out set, determining which checkpoint is the best becomes impossible. How can I decide which one to deploy? I could guess based on earlier experiments by assuming the best performance will occur after a similar number of training iterations, but looking over the earlier experiments, it's clear that the best performance doesn't consistently appear at the same point during training.