How to keep the test data from leaking into the training process of a machine learning algorithm?

Question

I read in many different sources that I need to split my data into a training set and a test set. Then I have to make sure that the algorithm is trained only on the training data, and do my best to keep the test data from leaking into the training process.

To avoid learning insignificant details of the data (which will increase the algorithm's generalization ability) I can further split my training data into the proper training set and a validation set multiple times, and choose the parameters of the algorithm that give the best average performance for all such splittings.

Finally I evaluate my algorithm on the test set, and get some numbers: MSE, RMSE, etc. But do these numbers really show, how good is my algorithm, and aren't they affected by the test set?

Of course, I didn't use my test data during training but the parameters of the algorithm that I got are valid only for this particular splitting into the training and test sets. If I split my data differently, I will get different MSE, RMSE, and different optimal parameters.

I can then do many different splittings into training/test, and calculate the error for each split. I that case how is it different from the cross-validation that I did during the first split? Do I need to further divide my training data into the proper data set and a validation set?

If I do many different splittings into the training an test sets, doesn't it mean that my test data leak into the training process because some data were in the test set for one splitting but in the training set for another splitting?

Answer 1

Just to clarify (and I think you've got this right, but I'm just being careful), it is best practice to:

1: Split your data into train and test

2: Split train into train and eval

3: Grid search over hyperparameters, for each combination, train on train, evaluate on eval. Select the hyperparameters which allow you to get the best score on the eval set

4: Using the best model (as ascertained in step 3), calculate the test loss, using the test data set create in step 1. Use your answer in step 4 to get the best estimate of how well your model will perform in production.

To be very clear, step 4 is not about hyperparameter selection, the hyperparameters are fixed by step 3. Step 4 is all about getting a better estimate of production performance, as what you get from step 3 is likely to be overly optimistic, as the particular hyperparameter combination used to overly optimised (overfit) to that particular train-eval split.

You can repeat steps 1-4 many times, but what you should not do, is try to use this repetition in any way, to find the best hyperparameters, you are using it to answer the question "how well is my model likely to perform on unseen data". For example, you might run it ten times and get a spread of accuracies from 65%-67%. You can then tell your boss, that if you deployed this model in production, it would likely have an accuracy of 65%-67% (or you're a bit more quantitative and you say it's the mean $\pm$ the standard error)

This leaves the question, "how do I select my best hyperparameters?". After all, you've run steps 1-4 many times and step 3 has given you a different best hyperparameter combination every time.

This is where you need to run step 3 one more time, but slightly differently. This time, you don't need a train-test split. You simply take all of your data, and do a train-eval split. You then run one further hyperparameter grid search, and that's the model you use in prod (or you could combine the train and eval sets, and train one more time, using those hyperparamters, depending on whether you're using a model that needs an eval set for early stopping like a neural net/xgboost or one that does not like a Random Forest). However, at this stage, you ignore the accuracy/peformance metrics that the training procedure is telling you. You already estimated your production accuracy/performance when you ran steps 1-4 multiple times, and that was the point of doing this.

I hope this answers your question, both on how to do the splitting, as well as illustrates why test data at no point leaks into the training data