4
$\begingroup$

I'm a little unclear about the expected use/value of a test set in machine learning. Here is a story that explains my confusion, assuming you're using a train/dev/test split:

You use your dev set to choose the best hyperparameters and make various tweaks, and when you're finally "done" you evaluate it on your test set. Your test performance comes back much worse than your dev performance.

So now you conclude, "My dev set must be too small, causing me to overfit my hyperparameters." So you make your dev set bigger, find new hyperparameters, and evaluate on your test set again. Now your dev and test performances are close to each other.

But note that you used your test set twice in that case. So in some sense you were fitting your hyperparameters to your test set, and it became a second development set.

To try to answer my own question: I guess you could say that the value of the test set is that without it, you would have never known you were overfitting your hyperparameters. And as long as we aren't using the test set "too much" in the above way (increasing the dev set size in a cycle again and again), it is still "mostly" unbiased. However we would have to concede that the test set is only truly unbiased if we only need it once.

Do you think this is an accurate take?

Incidentally, I'm unsure if there's anything else you can do (other than increasing dev set size) if your test performance comes back much worse. Well, I suppose you can cry ;). But are there any other options?

$\endgroup$
3
$\begingroup$

You're absolutely right and yes, it is actually possible to overfit to your validation data if you're not careful. Some researchers at google published an interesting article about this problem and a way to address it called the Reusable Holdout. The general idea is that you only access the test set through a special intermediary algorithm. Obviously this isn't how most people work though. In practice, I think a common approach is to use several holdouts: use one for most of your prototyping, and then once you're satisfied you can extend your evaluation to one or more of your additional holdouts.

$\endgroup$
1
$\begingroup$

actually you are right, you have to make use of the test set the less times you can if you want to be quite sure it will stay a test set, and not another dev set.

Said that, when you see that the performances on the test set are much worse than the dev set, you have other options than touch the size of the dev set:

1) add more penalties in the model, like a form of regularization (L1, L2, etc.);

2) decrease the degrees of freedom of your model, like the number of layer/nodes in case of NN, or the number of trees in case of decision trees et similia;

3) decrease the learning epochs;

4) try some (more aggressive) data augmentation technique;

You can also use data augmentation in the test set: for each sample of the test set you can use the same form of data augmentation you used on the training set, and finally taking as result the average of the various single results. Even if this technique do not resolve a very big performance difference from the dev set and the test set, it can make your model acceptable. An in this last case, you have of course to use it also on the real data when the model is deployed.

And in general if you are in need of accessing the test set several times, in order to have a less biased model you can reshuffle all the data, and then divide it again in train/dev/test. Of course, the rule of accessing the test set the less times possible stays true.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.