On this page of Google's Machine Learning Crash Course, we find the following statement:

"Test sets and validation sets "wear out" with repeated use. That is, the more you use the same data to make decisions about hyperparameter settings or other model improvements, the less confidence you'll have that these results actually generalize to new, unseen data. Note that validation sets typically wear out more slowly than test sets."

I don't get the last sentence. Why would a validation set wear out slower than a test set? To me it would wear out faster as the validation set is used a lot more frequently than the test set. What am I missing?

  • $\begingroup$ Good question! Wish Google allowed posting questions or indicated author for these posts. For all we know it could be a mistake in their writing. $\endgroup$ – hssay Aug 23 '18 at 1:41

It is difficult to say without access to the original author. However, I expect this refers to the ability of using each set to realise its purpose.

A validation set's purpose is to select hyperparameters that perform the best according to some metric. The best measurement on the validation set should always have the highest expectation of being the best in reality. If you make very many measurements, then the absolute probability of the best measurement being the real best could be low, but the chances of a generally poorly performing set of hyperparameters winning overall do not increase as fast. You can be reasonably certain that you have picked "one of the best" plus "the one with highest probability of being the best" even though that might be e.g. just a 10% chance if you have run 100s of validations.

A test set's purpose is to measure a metric without bias. If you use this for model comparison or selection, then this can be affected by maximisation bias - because there is uncertainty in the measurement, focusing on the relative values and picking a "best" almost certainly over-estimates the true value. This effect happens very quickly. If you measure metrics for two sets of hyperparameters and pick the best one, you should already expect that the value you got for the metric is an over-estimate. Note you still expect on average that you have picked the better option, but you cannot trust the measurement as much.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.