Is it always better to use the whole dataset to train the final model?

Question

A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, e.g. a product.

My question is: Is it always for the best to do so? What if the performance actually deteriorates?

For example, let us assume a case where the model scores around 65% in classifying the testing subset. This could mean that either the model is trained insufficiently OR that the testing subset consists of outliers. In the latter case, training the final model with them would decrease its performance and you find out only after deploying it.

Re-phrasing my initial question:

If you had a one-time demonstration of a model, such as deploying it on embedded electronics on-board an expensive rocket experiment, would you trust a model that has been re-trained with the test subset in the final step without being re-tested on its new performance?

Answer 1

A point that needs to be emphasized about statistical machine learning is that there are no guarantees. When you estimate performance using a held-out set, that is just an estimate. Estimates can be wrong.

This takes some getting used to, but it's something you're going to have to get comfortable with. When you say "What if the performance actually deteriorates?", the answer is sure, that could happen. The actual performance could be worse than you estimated/predicted. It could also be better. Both are possible. That's unavoidable. There is some inherent, irreducible uncertainty.

When you evaluate performance using a held-out test set, you are using data from the past to try to predict future performance. As they say, past performance is no guarantee of future results. This is a fact of life that we just have to accept.

You can't let this immobilize you. The fact that it's possible to do worse than you predicted is not a reason to avoid deploying to production a model trained on the data. In particular, it's also possible to do poorly if you don't do that. It's possible that a model trained on all the data (train+validation+test) will be worse than a model trained on just the train+validation portion. It's also possible that it will be better. So, rather than looking for a guarantee, we have to ask ourselves: What gives us the best chance of success? What is most likely to be the most effective?

And in this case, when you want to deploy to production, the best you can do is use all the data available to you. In terms of expected performance, using all of the data is no worse than using some of the data, and potentially better. So, you might as well use all of the data available to you to train the model when you build the production model. Things can still go badly -- it's always possible to get unlucky, whenever you use statistical methods -- but this gives you the best possible chance for things to go well.

In particular, the standard practice is as follows:

1. Reserve some of your data into a held-out test set. There is no hard-and-fast rule about what fraction to use, but for instance, you might reserve 20% for the test set and keep the remaining 80% for training & validation. Normally, all splits should be random.

2. Next, use the training & validation data to try multiple architectures and hyperparameters, experimenting to find the best model you can. Take the 80% retained for training and validation, and split it into a training set and a validation set, and train a model using the training set and then measure its accuracy on the validation set. If you are using cross-validation, you will do this split many times and average the results on the validation set; if you are not, you will do a single split (e.g., a 70%/30% split of the 80%, or something like that) and evaluate performance on the validation set. If you have many hyperparameters to try, do this once for each candidate setting of hyperparameter. If you have many architectures to try, do this for each candidate architecture. You can iterate on this, using what you've found so far to guide your choice of future architectures.

3. Once you're happy, you freeze the choice of architecture, hyperparameters, etc. Now your experimentation is done. Once you hit this point, you can never try any other options again (without obtaining a fresh new test set) -- so don't hit this point until you're sure you're ready.

4. When you're ready, then you train a model on the full training + validation set (that 80%) using the architecture and hyperparameters you selected earlier. Then, measure its accuracy on the held-out test set. That is your estimate/prediction for how accurate this modelling approach will be. You get a single number here. That number is what it is: if you're not happy with it, you can't go back to steps 1 and 2 and do more experimentation; that would be invalid.

5. Finally, for production use, you can train a model on the entire data set, training + validation + test set, and put it into production use. Note that you never measure the accuracy of this production model, as you don't have any remaining data for doing that; you've already used all of the data. If you want an estimate of how well it will perform, you're entitled to use the estimated accuracy from step 4 as your prediction of how well this will perform in production, as that's the best available prediction of its future performance. As always, there are no guarantees -- that's just the best estimate possible, given the information available to us. It's certainly possible that it could do worse than you predicted, or better than you predicted -- that's always true.

Answer 2

I personally haven't seen that for products going into production, but understand the logic.

Theoretically, the more data your deployed model has seen, the better is should generalise. So if you trained the model on the full set of data you have available, it should generalise better than a model which only saw for example train/val sets (e.g. ~ 90%) from the full data set.

The problem with this (and the reason we split data into train/val/test sets in the first place!) is that we want to be able to make statistical claims as to the accuracy on unseen data. As soon as we re-train a model again on all the data, it is no longer possible to make such claims.

[Edit]

Here is a related question on Cross-Validated, where the accepted answer makes similar points to me and mentions other ways of doing things.

---

We loop over:

1. train a model
2. assess performance on validation set $\rightarrow$ if satisfactory, go to step 5
3. change model
4. go to step 1
5. assess performance on test set
6. Present model with test accuracy found in step 5

Eventually, if you manage to get a great score on the test set, you can claim it generalises well. So the question as to whether re-training on the full dataset will improve performance on future unseen data is not strictly something you can test. Empirical evidence of better performance in other related problem sets would be the only source or guidance at the point in time when you must make the decision.

A sanity check would be to test the final re-trained model again on the original test set; expecting that it scores higher than it ever did when the model only saw the train/val set, because it has actually seen the test set during training. This wouldn't make me feel 100% confident that this final model is superior in all future cases, but at least it is as good as it can be with the given data.

Perhaps there are more rigorous arguments against doing what you say (probably academically motivated), however it does seem appealing for practical applications!

Answer 3

Once you have obtained optimal hyperparamters for your model, after training and cross validating etc., in theory it is ok to train the model on the entire dataset to deploy to production. This will, in theory, generalise better.

HOWEVER, you can no longer make statistical / performance claims on test data since you no longer have a test dataset.

If you deploy a model to production using the entire training dataset, and you know the true values of the target variable of the new incoming data (i.e the data the production model is making predictions on), then you can calculate real time performance metrics as this new data is like test data (it was unseen to the model during training). From this process you could update the models hyperparameters to achieve better performance.

But if you knew the target values of new data, why would you train a model in the first place?

In general, I would say if you have enough data with enough variety, then shuffling and splitting the data 80:20 training:test should be sufficient to train a robust model and not have to worry about generalisation issues (assuming of course you regularize the model).

Answer 4

One of the reasons of having a data set is to avoid overfitting. If you employ cross-validation, you essentially allow the entire dataset to act as the training set, but retraining won’t let you validate whether there is sign of overfitting. I guess that either way (cross validation or retrain with the entire data set) should not dramatically change your result (from my uneducated guess), but you won’t be able to do hyperparameter tuning or validate your model performance as you don’t have a test set. Whether it ends up being better, it is hard to say, but I guess the only way to know is to do an A/B of the two models over real data over time.

Answer 5

Unless you're limiting yourself to a simple class of convex models/loss functions, you're considerably better off keeping a final test split. Here's why:

Let's say you collect iid sample pairs from your data generating distribution, some set of (x, y). You then split this up into a training and test set, and train a model on the training set. Out of that training process you get a model instance, f(x; w). Where w denotes the model parameters.

Let's say you have N observations in the test set. When you validate this model on that test set you form the set of test predictions, {f(x_i, w) : i=1,2,...,N} and compare it to the set of test labels {y_i : i=1,2,...,N} using a performance metric.

What you're able to say using N independent observations is how you expect that model instance, i.e. the function given a specific w, will generalize to other iid data from the same distribution. Importantly, you only really have one observation (that w you found) to comment on your process for determining f(x, w), i.e. the training process. You can say a little more using something like k-fold cross validation, but unless your willing to do exhaustive cross-validation (which is not really feasible in a vision or NLP context), you'll always have less data on the reliability of your training process.

Take a pathological example, where you draw the model parameters at random, and you don't train them at all. You obtain some model instance f(x, w_a). Despite the absurdity of your (lack of) training process, your test set performance is still indicative of how that model instance will generalize to unseen data. Those N observations are still perfectly valid to use. Maybe you'll have gotten lucky and have landed on a pretty good w_a. However, if you combine the test and training set, then "retrain" the model to obtaining a w_b, you're in trouble. The results of your previous test performance amounts to basically a point estimate of how well your next random parameter draw will fare.

There are statistical results that you can use to comment on the reliability of the entire training process. But they require some assumptions about your model class, loss function, and your ability to find the best f(x, w) from within that class for any given set of training observation. With all that, you can get some bounds on the probability that your performance on unseen data will deviate by more that a certain amount from what you measured on the training data. However, those results do not carry over (in a useful way) to overparameterized and non-convex models like modern neural networks.

The pathological example above is a little over the top. But as an ML researcher and consultant, I have seen neural network training pipelines that occasionally latch on to terrible local minima, but otherwise perform great. Without a final test split, you'd have no way of being sure that hadn't happened on your final retraining.

More generally, in a modern machine learning context, you cannot treat the models coming out of your training process as interchangeable. Even if they do perform similarly on a validation set. In fact, you may see considerable variation from one model to the next when using the full bag of stochastic optimization tricks. (For more details on that, check out this work on underspecification.)