1
$\begingroup$

Would this work?

We want to train a neural net.

We have 50 datapoints and want a split of 30 for train, 10 for validation, 10 for test.

We want to do 5-fold cross validation.

We use the following pseudo code to evaluate a configuration of hyperparameters.

For each of the 5 folds:
    Randomly split the 50 datapoints between train, val, test.
    While validation loss decreases:
        Train model on training dataset (training to minimise loss).
        Calculate model loss on validation dataset.

    Calculate model loss on test dataset and store in array test_losses_array.

Calculate the average of test_losses_array. This is the score for this configuration.

So each of the folds would have a different test dataset, which is where I believe this different from usual where the test dataset is fixed. Why is it better to fix the test dataset and not do what I have done? It seems like changing the test dataset each time reduces the risk of the test dataset simply being a test set that is easy to perform well on.

I guess we could use the same test dataset to evaluate all configurations, why is this better than what I propose (changing the test dataset every fold for every configuration)?

Also if there are any other problems with my training loop, please shout them out.

$\endgroup$

1 Answer 1

1
$\begingroup$

What you are trying to do is called Nested cross-validation. Nested cross-validation is an approach to model hyperparameter optimization and model selection that attempts to overcome the problem of overfitting the training dataset. Take a look at this post to learn more about it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.