3
$\begingroup$

Say I've divided the data into 3 parts: training, validation and test. I know for example, that in Neural Networks, the number of hidden layers is a hyper parameter. Why can't I train numerous NN architectures in the training set and then test their accuracy in the test set; thereby allowing me to choose one final model? What is the purpose of the validation set in this instance?

$\endgroup$
3
$\begingroup$

I wouldn't say you can not tune the hyper-parameters in the trainig dataset, but the purpose of doing so is different than in the validation set.

In general, what it is intended with a ML algorithm is how to optimally classify or perform regression given some training data. Once the model is trained, it will be used with new data to obtain predictions, thus:

  1. We need some training data. With this dataset, the model will look for the optimal weights and biases that minimizes the selected loss function.
  2. We need some upper bound to our performance metric: for example, we can decide that our model has to perform like a human, which has showed an average metric of 95% for some specific task. The upper bound can be the best score in a certan benchmark, etc.
  3. We start training the model on the training data and we evaluate its metric, lets take the accuracy. If the accuracy is too low, we can tune the hyper-paramters until the accuracy increases in the training data (no evaluation dataset is used here). The difference between our model's accuracy and the upper bound is taken as the bias of the model (this bias has not to be confused with the biases of the neurons). So, the accuracy on the training data gives us the bias of our model.
  4. Once we decided our bias is reasonable (optimally 0) arises the question of how the model will perform when it is fed with data that has not be used for training (the real application). The difference between the accuracy on this unseen data and the accuracy on the training data is the variance of the model. This unseen data is the validation set and give us an idea of how the model generalizes to new data. If the variance is high, then the model poorly generalizes to new data. Then, we perform hyper-paramater tunning and evaluating the accuracy on the validation data until the variance is low enough, trying to not worsening the bias (variance-bias trade-off).

Example: desired accuracy = 95%. Training accuracy = 93%. Validation accuracy = 82% -> Bias = 2%, Variance = 11%

Summarizing:

  • Tunning in the training data = decreasing bias
  • Tunning in the validation data = decreasing variance

What's more unclear is the role of the test dataset. I usually use it when I decided that the model will be no longer changed and I need the final metric that describes the model. The test set can also be used to have an idea about how the model performs with data which have not be intended to work with. In general, the test set gives the power of the model for the inference task.

$\endgroup$
1
  • $\begingroup$ Firstly, my comment depends on terminology so I hope people are aware of this. From the textbooks and materials I used, I prefer to define (inductive) bias as the "minimum risk achievable by a predictor in the hypothesis class". Therefore, tuning in the training data is really trying to decrease this bias. What is the minimum bias you can get? The error achieved by Bayes optimal classifier (like the upper bound above). In that sense, I will prefer not to call "The difference between our model's accuracy and the upper bound is taken as the bias". To conclude, nothing's wrong :) $\endgroup$ – LKS Mar 20 '19 at 5:58
1
$\begingroup$

This is exactly why. If you use your test dataset to select the best architecture that can predict, then you are not using it as a test data, you are using your test data as a validation data. And, in the end, you run out of data to test.

To make it clear:

  • Train dataset: Use your train dataset to learn the parameters of your model (for example, weight of connections in your NN).

  • Validation dataset: Use your validation dataset to see how they perform and tune these hyperparameters (for example, prune your nets, etc).

  • Test dataset: Your test dataset should only be used just in the last instance to test the model. Otherwise, you are not be able to test on data that you haven't use to train your model.

You can read more about it on this article.

$\endgroup$
3
  • 1
    $\begingroup$ I think the number of connections is not learnt, is something that can be set/tuned... what the network learns are the weights and biases $\endgroup$ – ignatius May 14 '18 at 21:50
  • $\begingroup$ @ignatus you are completely right, i have corrected it. $\endgroup$ – Federico Caccia May 14 '18 at 21:55
  • 1
    $\begingroup$ I appreciate you correction but... the hyper-parameter, whatever they are, are not learnt, are selected or tuned. Furthermore, the weights are not hyper-paramters. $\endgroup$ – ignatius May 14 '18 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.