0
$\begingroup$

Despite doing/using it a few times, I'm still slightly confused by the use of a validation set for hyper parameter tuning.

As far as I can tell, I choose a model, train it on training data, assess performance on training data, then do hyper parameter tuning assessing model performance on validation data, then choose the best model and test this on test data.

In order to do this, I basically need to pick a model at random for training data. What I don't understand is I don't know which model is going to be best at the start anyway. Let's say I think neural networks and random forests may be useful for my problem. So why don't I start searching with a general e.g. Neural Network architecture, random forest architecture, and from the very beginning, assess which model is best on a small portion of data varying all hyper parameters at the start anyway.

Basically why choose a human based "guess" to do the training, then hyperparameter tune in validation phase? Why not "start with total uncertainty", and do a broad search, assess performance of a wide range of hyperparameters from a general neural network or random forests or ... architecture, from the very beginning?

Thanks!

$\endgroup$

1 Answer 1

0
$\begingroup$

You perform hyperparameter tuning using train dataset. Validation dataset is used to make sure the model you trained is not overfit. The issue here is that the model has already "seen" the validation dataset and it is possible that the model doesn't perform as expected against new/unseen data. That's why you need an additional dataset, namely test dataset.

$\endgroup$
1
  • $\begingroup$ Hi, thanks. I don't quite understand. In the bit, e.g. "Training Set, Validation Set, and Holdout Set. What is a Training Set?" medium.com/@sanidhyaagrawal08/… In this example they give, I have chosen a model to do training on. But then hyper parameters are such an important part of a model, why do I leave them for just the validation section? I seem to have 'trained' my model, then after it started changing parameters. I don't get quite why as I can't H-Opt without this? $\endgroup$
    – Socorro
    Jun 9 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.