0
$\begingroup$

Despite doing/using it a few times, I'm still slightly confused by the use of a validation set for hyper parameter tuning.

As far as I can tell, I choose a model, train it on training data, assess performance on training data, then do hyper parameter tuning assessing model performance on validation data, then choose the best model and test this on test data.

In order to do this, I basically need to pick a model at random for training data. What I don't understand is I don't know which model is going to be best at the start anyway. Let's say I think neural networks and random forests may be useful for my problem. So why don't I start searching with a general e.g. Neural Network architecture, random forest architecture, and from the very beginning, assess which model is best on a small portion of data varying all hyper parameters at the start anyway.

Basically why choose a human based "guess" to do the training, then hyperparameter tune in validation phase? Why not "start with total uncertainty", and do a broad search, assess performance of a wide range of hyperparameters from a general neural network or random forests or ... architecture, from the very beginning?

Thanks!

$\endgroup$

3 Answers 3

0
$\begingroup$

You perform hyperparameter tuning using train dataset. Validation dataset is used to make sure the model you trained is not overfit. The issue here is that the model has already "seen" the validation dataset and it is possible that the model doesn't perform as expected against new/unseen data. That's why you need an additional dataset, namely test dataset.

$\endgroup$
2
  • $\begingroup$ Hi, thanks. I don't quite understand. In the bit, e.g. "Training Set, Validation Set, and Holdout Set. What is a Training Set?" medium.com/@sanidhyaagrawal08/… In this example they give, I have chosen a model to do training on. But then hyper parameters are such an important part of a model, why do I leave them for just the validation section? I seem to have 'trained' my model, then after it started changing parameters. I don't get quite why as I can't H-Opt without this? $\endgroup$
    – Socorro
    Commented Jun 9, 2022 at 21:28
  • $\begingroup$ @Socorro You don't train some random single model on the training set. You train a lot of (model&)hyperparameter( pair)s on the training set (learning parameters) and evaluate them on the validation set to select "the best". $\endgroup$
    – Ben Reiniger
    Commented Dec 5 at 23:11
-1
$\begingroup$

There are a few reasons why hyperparameter tuning is typically done on the validation set rather than on the training set or on a small portion of the data at the very beginning:

  1. Overfitting: If you tune the hyperparameters on the training set, the model may end up overfitting to the training data. This means that the model may perform very well on the training data but may not generalize well to new, unseen data. On the other hand, if you tune the hyperparameters on the validation set, the model will be less likely to overfit because the validation set is different from the training set and is used to evaluate the model's performance on unseen data.

  2. Bias: If you tune the hyperparameters on a small portion of the data at the very beginning, the model may be biased towards that small portion of the data and may not generalize well to the rest of the data. This is because the model may have learned some patterns or features that are specific to the small portion of the data and may not be representative of the entire dataset.

  3. Efficiency: Tuning the hyperparameters on the validation set is generally more efficient than tuning them on the training set or on a small portion of the data at the very beginning. This is because the validation set is typically larger than the training set, and tuning the hyperparameters on a larger dataset allows the model to learn more robustly and generalize better.

In short, hyperparameter tuning is typically done on the validation set because it helps to prevent overfitting, reduces bias, and is more efficient than other approaches.

$\endgroup$
-1
$\begingroup$

The primary reason for this approach lies in the need to prevent overfitting and accurately assess the generalization performance of the model. Let's delve into the details:

  1. Preventing Overfitting: Overfitting occurs when a model learns to capture noise and patterns specific to the training dataset, resulting in poor performance on unseen data.

  2. Generalization Performance: The ultimate goal of machine learning models is to perform well on unseen data. By evaluating the model's performance on a validation dataset, we gain insights into how well it generalizes to new, unseen samples.

  3. Avoiding Data Leakage: Tuning hyperparameters on the training dataset itself can lead to data leakage, where information from the validation or test set inadvertently influences the model selection process.

In conclusion, hyperparameter tuning on a validation dataset, rather than at the very beginning of model training, helps prevent overfitting, assess the model's generalization performance accurately, avoid data leakage, and enables iterative improvement of the model's hyperparameters. This approach ultimately leads to better-performing and more robust machine learning models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.