When using K-Fold CV, is it still useful to have a Train/Validation/Test split?
Or simply just a Train/Test? I.e. split up data into k bins, and leave one out for testing, train on the rest, and take average of the scores.
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
It depends. If you're evaluating your model's performance without tuning hyperparameters, then a train/test split is sufficient.
If you're tuning hyperparameters, then you need a validation set. Within each fold, you'll train on the training set (of course), using the validation set to tune hyperparameters. Then you'll evaluate performance on the test set.
-
$\begingroup$ So if using a validation set, and we split the data into 5 groups, there would be 3 for training, 1 for validation, and 1 for testing? $\endgroup$– ShinobiiCommented Dec 18, 2019 at 4:15
-
1$\begingroup$ That's one way to do it. You can also think of it as having 4 pieces for training+validation, and 1 piece for testing. Then you can split the training+validation set (the 4 pieces), in whichever way seems appropriate. Does that make sense? $\endgroup$– zachdjCommented Dec 18, 2019 at 4:19
-
1$\begingroup$ Yep, perfectly clear. What seemed like such a simple question, was hard to find in all the literature I have been reading. Thanks for your response and understanding. $\endgroup$– ShinobiiCommented Dec 18, 2019 at 19:20
$\begingroup$
$\endgroup$
Very open ended and context dependent, but generally:
You are already have a test set, and actually a "new" one in every CV-fold.
Do your model experimentations with CV, when you are done train on the whole dataset at once.
answered Dec 17, 2019 at 23:12