Can using model evaluation metrics to choose a model cause data leakage?

Question

I was reading a blog post about improving machine-learning model train/validate/test splits. Towards the end was this remark:

I say we should be more creative in the way we test machine learning models than a 60-20-20 split. But there is a catch – if I see the results of tests and decide this model isn’t good enough, then what do I do? If I let the information of a test set to change my decision of which model to use, then it’s not really an unbiased test anymore – this is what people called “peeking” or “data leakage.” In general, I would say one should avoid this as much as possible. But the more tests you put on the model, the more likely you will see some failed tests and as a result, peek. Now we have a dilemma.

I found this a bit strange, because my impression is this is precisely how the test is used, to determine whether or not to use a model. However, it did occur to me that maybe what the author is getting at, is something like this:

There are many possible models one can use for a given dataset, some of which may by chance perform better on evaluation metrics than others, even when you withhold the test set from them during training. By using a large ensemble of models, and selecting the one with the best performance, you are increasing the likelihood that this performance was due to chance rather than the model truly representing/predicting the data well. Basically, similar to the idea behind p-hacking in scientific studies.

It seems to me that in practicality, this is probably an issue of degree (if you are testing 2-5 models, it may not be as much of a problem as if you test hundreds or more), but I can't find much discussion about this outside of this blog post.

I could easily be missing something here, but I want to ask, can/does using test evaluation metrics to perform model selection introduce a kind of data leakage (leak information about the test data to the model)? Are there any more authoritative sources on this issue?

Answer 1

This is exactly the same reason why hyper-parameter tuning, feature selection and other decisions which impacts the final model should be done on a validation set, not on the final test set.

In theory one should really evaluate the model only once on a fresh test set, so if there is a chance that a model won't be the final one depending on its performance then it should be evaluated on another validation set. The training data can be split as many times as needed (assuming it's large enough) in order to have enough validation sets.

Clearly things are not always done perfectly in practice, and this can be considered acceptable to some extent. For example the repeated usage of benchmark datasets, which is really useful in order to compare models, is potentially a source of data leakage: indirectly, people use the knowledge based on the evaluation of the last state of the art model in order to design a new competitive model.