2
$\begingroup$

For hyperparameter optimization I see two approaches:

  1. Splitting the dataset into train, validation and test, and optimize the hyperparameters based on the results of training on the train dataset and evaluating on the validation dataset, leaving the test set untouched for final performance estimation.

  2. Splitting the dataset into train and test, and optimize the hyperparameters using crossvalidation on the train set, leaving the test set untouched for final performance estimation.

So which approach is better?

$\endgroup$
3
$\begingroup$

The $k$-fold cross-validation (CV) process (method 2) actually does the same thing as method 1, but it repeats the steps on the training and validation sets $k$ times. So with CV the performance is averaged across the $k$ runs before selecting the best hyper-parameter values. This makes the performance and value selection more reliable in general, since there is less risk to obtain the best result by chance. However it takes much longer (since repeating $k$ times), so if the training process is long it's not always practical to use CV.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.