10
$\begingroup$

Consider a neural network:

For a given set of data, we divide it into training, validation and test set. Suppose we do it in the classic 60:20:20 ratio, then we prevent overfitting by validating the network by checking it on validation set. Then what is the need to test it on the test set to check its performance?

Won't the error on the test set be somewhat same as the validation set as for the network it is an unseen data just like the validation set and also both of them are same in number?

Instead can't we increase the training set by merging the test set to it so that we have more training data and the network trains better and then use validation set to prevent overfitting? Why don't we do this?

$\endgroup$
  • 3
    $\begingroup$ You'd like it to be the same but you can't be sure because you've touched it for hyperparameter optimization and early stopping, so you need a virgin test set. $\endgroup$ – Emre Apr 13 '17 at 20:01
  • $\begingroup$ @Emre But the weights will get adjusted based on the training set and not on the validation set, so the result on test and validation set shouldn't be way to different. $\endgroup$ – user1825567 Apr 14 '17 at 4:24
  • $\begingroup$ No, they don't (get adjusted according to the training set). That's for regular parameters. $\endgroup$ – Emre Apr 14 '17 at 4:36
14
$\begingroup$

Let's assume that you are training a model whose performance depends on a set of hyperparameters. In the case of a neural network, these parameters may be for instance the learning rate or the number of training iterations.

Given a choice of hyperparameter values, you use the training set to train the model. But, how do you set the values for the hyperparameters? That's what the validation set is for. You can use it to evaluate the performance of your model for different combinations of hyperparameter values (e.g. by means of a grid search process) and keep the best trained model.

But, how does your selected model compares to other different models? Is your neural network performing better than, let's say, a random forest trained with the same combination of training/test data? You cannot compare based on the validation set, because that validation set was part of the fitting of your model. You used it to select the hyperparameter values!

The test set allows you to compare different models in an unbiased way, by basing your comparisons in data that were not use in any part of your training/hyperparameter selection process.

$\endgroup$
7
$\begingroup$

The test set and cross validation set have different purposes. If you drop either one, you lose its benefits:

  • The cross validation set is used to help detect over-fitting and to assist in hyper-parameter search.

  • The test set is used to measure the performance of the model.

You cannot use the cross validation set to measure performance of your model accurately, because you will deliberately tune your results to get the best possible metric, over maybe hundreds of variations of your parameters. The cross validation result is therefore likely to be too optimistic.

For the same reason, you cannot drop the cross validation set and use the test set for selecting hyper parameters, because then you are pretty much guaranteed to be overestimating how good your model is. In the ideal world you use the test set just once, or use it in a "neutral" fashion to compare different experiments.

If you cross validate, find the best model, then add in the test data to train, it is possible (and in some situations perhaps quite likely) your model will be improved. However, you have no way to be sure whether that has actually happened, and even if it has, you do not have any unbiased estimate of what the new performance is.

From witnessing many Kaggle competitions, my experience is that tuning to the test set by over-using it is a real thing, and it impacts those competitions in a large way. There is often a group of competitors who have climbed the public leaderboard and selected their best model in test (the public leaderboard is effectively a test set), whilst not being quite so thorough on their cross validation . . . these competitors drop down the leaderboard when a new test set is introduced at the end.

One approach that is reasonable is to re-use (train + cv) data to re-train using the hyper-params you have found, before testing. That way you do get to train on more data, and you still get an independent measure of performance at the end.

If you want to get more out of cross validation, the usual approach is k-fold cross validation. A common trick in Kaggle competitions is to use k-fold cross validation, and instead of re-combining the data into a larger (train + cv) training set, to ensemble or stack the cv results into a meta-model.

Finally, always check that your splits for validation and test are robust against possible correlation within your data set.

$\endgroup$
  • 1
    $\begingroup$ what do you mean by "robust against possible correlation within your data set"? $\endgroup$ – user6903745 Apr 13 '18 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.