0
$\begingroup$

Suppose I have 100 samples, then I want to use 5-fold CV, is the ratio of training set: validation set: testing set is 84:16:16? Is the number of data in validation set should always equal to testing set?

$\endgroup$
0
$\begingroup$

To answer this we need to understand what is the purpose of the all kinds of sets:
-Training set -we use it for learning purpose.Our model learn from it
-Validation set-we use to tune the parameters (for example to choose the number of hidden units in the NN).We do not learn from it.
-Test set -we use it to evaluate our model.We don`t learn from it
Many a times, people first split their dataset into 2 — Train and Test. After this, they keep aside the Test set, and randomly choose X% of their Train dataset to be the actual Train set and the remaining (100-X)% to be the Validation set. My personal preferences are 70-10-20,but it depends how much data and what kind of data do you have .

$\endgroup$
3
  • $\begingroup$ Hi, I always heard that train:val:test should be equal to 60:20:20 or 80:10:10 or 70:15:15. Sounds like the validation set and test set should be equal size, is that true? $\endgroup$ – sherry xu Dec 29 '20 at 3:40
  • $\begingroup$ @sherryxu -the train-test-validation split ratio is also quite specific to your use case and it is not necessary validation and test set to be the same .For more detail explanation please check towardsdatascience.com/… $\endgroup$ – mariq vlahova Jan 1 at 11:46
  • $\begingroup$ Many thanks! I got that.:) $\endgroup$ – sherry xu Jan 2 at 10:36
0
$\begingroup$

For K-fold cross validation, you would not need a separate validation set. In your example of 100 samples, if you have train:test set ratio to be as 80:20 and follow the 5-fold CV algorithm, the train set (80 samples) will be split into five 16 sample subsets ($s_i$) as in $\{s_1, s_2, s_3, s_4, s_5\}$. At each iteration, the model will select any 4 subsets out of the 5 subsets, based on your selection criteria, and use the left-out set as the validation set.

Once the training is completed, you can test the performance of the final model with the held-out test set (20 samples) to get an unbiased estimate of the performance of the model.

It is not necessary to have same number of samples in validation set and test set.

$\endgroup$
3
  • $\begingroup$ Many thanks for the reply. But in the example, you seem treated the validation set as the test set, is it correct? $\endgroup$ – sherry xu Oct 16 '20 at 13:35
  • $\begingroup$ Yes, in K-fold cross validation, validation set plays the role of a test set within each iteration. $\endgroup$ – cmn Oct 16 '20 at 17:21
  • $\begingroup$ Thank you so much :) $\endgroup$ – sherry xu Dec 28 '20 at 10:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.