Why use both validation set and test set?

Question

Consider a neural network:

For a given set of data, we divide it into training, validation and test set. Suppose we do it in the classic 60:20:20 ratio, then we prevent overfitting by validating the network by checking it on validation set. Then what is the need to test it on the test set to check its performance?

Won't the error on the test set be somewhat same as the validation set as for the network it is an unseen data just like the validation set and also both of them are same in number?

Instead can't we increase the training set by merging the test set to it so that we have more training data and the network trains better and then use validation set to prevent overfitting? Why don't we do this?

Answer 1

Let's assume that you are training a model whose performance depends on a set of hyperparameters. In the case of a neural network, these parameters may be for instance the learning rate or the number of training iterations.

Given a choice of hyperparameter values, you use the training set to train the model. But, how do you set the values for the hyperparameters? That's what the validation set is for. You can use it to evaluate the performance of your model for different combinations of hyperparameter values (e.g. by means of a grid search process) and keep the best trained model.

But, how does your selected model compares to other different models? Is your neural network performing better than, let's say, a random forest trained with the same combination of training/test data? You cannot compare based on the validation set, because that validation set was part of the fitting of your model. You used it to select the hyperparameter values!

The test set allows you to compare different models in an unbiased way, by basing your comparisons in data that were not use in any part of your training/hyperparameter selection process.

Answer 2

The test set and cross validation set have different purposes. If you drop either one, you lose its benefits:

• The cross validation set is used to help detect over-fitting and to assist in hyper-parameter search.

• The test set is used to measure the performance of the model.

You cannot use the cross validation set to measure performance of your model accurately, because you will deliberately tune your results to get the best possible metric, over maybe hundreds of variations of your parameters. The cross validation result is therefore likely to be too optimistic.

For the same reason, you cannot drop the cross validation set and use the test set for selecting hyper parameters, because then you are pretty much guaranteed to be overestimating how good your model is. In the ideal world you use the test set just once, or use it in a "neutral" fashion to compare different experiments.

If you cross validate, find the best model, then add in the test data to train, it is possible (and in some situations perhaps quite likely) your model will be improved. However, you have no way to be sure whether that has actually happened, and even if it has, you do not have any unbiased estimate of what the new performance is.

From witnessing many Kaggle competitions, my experience is that tuning to the test set by over-using it is a real thing, and it impacts those competitions in a large way. There is often a group of competitors who have climbed the public leaderboard and selected their best model in test (the public leaderboard is effectively a test set), whilst not being quite so thorough on their cross validation . . . these competitors drop down the leaderboard when a new test set is introduced at the end.

One approach that is reasonable is to re-use (train + cv) data to re-train using the hyper-params you have found, before testing. That way you do get to train on more data, and you still get an independent measure of performance at the end.

If you want to get more out of cross validation, the usual approach is k-fold cross validation. A common trick in Kaggle competitions is to use k-fold cross validation, and instead of re-combining the data into a larger (train + cv) training set, to ensemble or stack the cv results into a meta-model.

Finally, always check that your splits for validation and test are robust against possible correlation within your data set.

Answer 3

In theory (an ideal world)

• Training set: The model's many (e.g. 10 million) parameters are directly tuned using the training set, typically via gradient descent.
• Validation set: The model's few (e.g. 100) hyperparameters (e.g. number of epochs, number of layers, number of channels) are indirectly tuned using the validation set.
• Test/inference set: The final model is evaluated once and only once at the very end of the scientific process, for the sole purpose of reporting its true performance. No further models are developed after this point, since otherwise, the researcher will have used the test set in the process of model training/selection.

In other words, a validation set is just a secondary training set. This is completely different from the function of the test set, which is never used during model training/design/selection!

In practice (real life)

There are some common standardized datasets with public test sets. Researchers measure their model performance on these, and then report their results. Researchers are strongly motivated to create state-of-the-art (SOTA) models with the best performance. Thus, there is a strong incentive to measure their model performance on the test set at some point in the middle of the model design process. Researchers that have iron wills and are committed to an authentic scientific process are at a disadvantage against those who have tuned their models on the test set.

However, even a community of diligent well-intentioned researchers will commit grave errors. For instance, merely reading existing "100% authentic literature" that has been measured on the test set will influence a reader's model design/selection process! A researcher might choose to build upon a previous SOTA model, or use previous SOTA techniques. The degree to which the test set has an influence on the model design process varies, but any amount of influence immediately colors the process and puts different researchers on uneven footing!

In reality, what most researchers use is:

• Training set: The model's many (e.g. 10 million) parameters are directly tuned using the training set, typically via gradient descent.
• Primary validation set: The model's few (e.g. 100) hyperparameters (e.g. number of epochs) are indirectly tuned using the primary validation set.
• Secondary standardized validation set: Often incorrectly referred to as the "test set". The model is evaluated intermittently at various stages during the design process, or at least once in prior works, for the "unscientific" purpose of improving its reported performance within a paper.

An alternative: hidden test sets

It is nearly impossible for researchers to live in an information vacuum, or have completely iron wills. Thus, one common alternative is for an external unaffiliated third-party to make a single annual assessment of the trained model performance on a hidden test set. A new test set must be created for use every year, after all model results are simultaneously published. This is the method used by challenge competitions, and it is the only real approximation for fulfilling the requirements of the theoretical test set listed above.

Some competitions (e.g. ILSVRC) allow reuse of the test set along shorter periods (e.g. 2 submissions per week). However, even this has been exploited by researchers to cheat:

The Challenge rules state that you must only test your code twice a week, because there’s an element of chance to the results.

Baidu has admitted that it used multiple email accounts to test its code roughly 200 times in just under six months – over four times what the rules allow.

MIT Technology Review