0
$\begingroup$

I am following this Google's series: Machine Learning Crash Course.

On the chapter about generalisation, they make the following statement:

Good performance on the test set is a useful indicator of good performance on the new data in general, assuming that:

- The test set is large enough.

- You don't cheat by using the same test set over and over.

Why exactly is the second point a bad one? As long as one does not use the test set for the training stage, why is it bad to keep using the same test set to test a model's performance? It's not like the model will get a bias by doing so (the test set is not updating any of the model's parameters internally).

$\endgroup$
  • $\begingroup$ Because we won't be generalising it then, Statistics about the data might change from the very next day... And our model won't be able to Extrapolate the results and give us rather worse results... To keep it upto date with the data at hand.. otherwise do you think that your predictions will vary a lot if you keep on using the same data to benchmark.. $\endgroup$ – Aditya Jul 20 '18 at 4:25
1
$\begingroup$

You don't cheat by using the same test set over and over.

Correct point! Test set is for testing the performance of model, how good your model performs on test data by learning information from training data. There is no problem in doing this, until you try to create a new model or increase the performance of already-built model using test set. Then it will introduce bias in your model, as it will also try to learn the information in test set. In this case, you will begin to optimize your model on test set also. Even if you are not training your model on test set, it will happen.

As given in this answer,

Performance estimates should always be done on completely independent data. If you are optimizing some aspect based on test data, then your test data is no longer independent and you would need a validation set.

So even if test set does not update your model parameters, test set will no longer be unseen data and its purpose will end. Then your test set will become validation set (as in cross-validation), and you would not be left with any actual test set.

$\endgroup$
1
$\begingroup$

In fact this will introduce some bias to your model because you will distinctly choose a model that performs well on the testing set. This is indeed what bias consists of. Ideally you would want to have a training set, a validation set and a testing set. You will only use the testing set right before writing the spec of your model. If the performance is much poorer than expected at this very last step, then you need to restart from the top. Get new data to reduce "the polluting" of the testing set and retrain and validate until you are satisifed then use the testing set.

Consider if your testing set does not generalize well on the actual distribution, this may be improbable but it is possible. Then a good model will perform poorly on this test set, however it would otherwise be a good model if you did not bias your decision using your test set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.