I was reading a blog post about improving machine-learning model train/validate/test splits. Towards the end was this remark:
I say we should be more creative in the way we test machine learning models than a 60-20-20 split. But there is a catch – if I see the results of tests and decide this model isn’t good enough, then what do I do? If I let the information of a test set to change my decision of which model to use, then it’s not really an unbiased test anymore – this is what people called “peeking” or “data leakage.” In general, I would say one should avoid this as much as possible. But the more tests you put on the model, the more likely you will see some failed tests and as a result, peek. Now we have a dilemma.
I found this a bit strange, because my impression is this is precisely how the test is used, to determine whether or not to use a model. However, it did occur to me that maybe what the author is getting at, is something like this:
There are many possible models one can use for a given dataset, some of which may by chance perform better on evaluation metrics than others, even when you withhold the test set from them during training. By using a large ensemble of models, and selecting the one with the best performance, you are increasing the likelihood that this performance was due to chance rather than the model truly representing/predicting the data well. Basically, similar to the idea behind p-hacking in scientific studies.
It seems to me that in practicality, this is probably an issue of degree (if you are testing 2-5 models, it may not be as much of a problem as if you test hundreds or more), but I can't find much discussion about this outside of this blog post.
I could easily be missing something here, but I want to ask, can/does using test evaluation metrics to perform model selection introduce a kind of data leakage (leak information about the test data to the model)? Are there any more authoritative sources on this issue?