I have a dataset of 39 medical MR images, and I have to build a model to classify the tumor type. so is it suitable to use k-fold cross validation for validating the model? if so, what would be the number of K?
5
-
2$\begingroup$ LOOCV (k=1) is best given your data. $\endgroup$– user2974951Dec 5, 2018 at 14:00
-
1$\begingroup$ thank you sir for your quick reply, so it is not recomended to use k-fold cross validation? $\endgroup$– ginDec 5, 2018 at 14:12
-
1$\begingroup$ LOOCV is a type of CV when k=n (not k=1 as in my previous comment), so they are similar, but LOOCV will be better in this case since you have a very small dataset and k-fold CV will partion it even further into smaller datasets. $\endgroup$– user2974951Dec 5, 2018 at 14:19
-
1$\begingroup$ you mean by n is the data samples? in my case 39? $\endgroup$– ginDec 5, 2018 at 15:39
-
1$\begingroup$ Yes, n is sample size. $\endgroup$– user2974951Dec 6, 2018 at 6:30
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
Given the size of your data set, the best approach to cross validation is the Leave-One-Out method. You haven't discussed the language or package you used for your model, but generally speaking you set the $k$ equal to the number of records. In your case, that 39. This will cause your model to train on 38 instances and predict the 39th, with each instance eventually will receive a classification.
-
1$\begingroup$ Thank you sir for your reply, I am using matlab. so you're suggesting using Leave-One-Out method with k=39? $\endgroup$– ginDec 5, 2018 at 14:14
-
1$\begingroup$ I am not sure about MATLAB, but if it has a generic Cross Validation function, you should probably set $k=39$. $\endgroup$– SkiddlesDec 5, 2018 at 14:15
-
1$\begingroup$ Did that work for you? I have found that sometimes the measures get out of whack with LOO. It could have been user error, but I just didn't have time to work through it so changed $k$ to something smaller. Of course, as the number of instances increases, using LOO takes a lot longer to complete and it is questionable if a smaller $k$ is any less valuable overall. You really wouldn't want to try LOO with a sizable data set. $\endgroup$– SkiddlesDec 5, 2018 at 15:13
-
1$\begingroup$ No I haven't tried. to be honest, I haven't started building the system yet. I just wanted to gather information before starting. so you mean that i should make k smaller than 39 and try the best outcome? $\endgroup$– ginDec 5, 2018 at 15:42
-
1$\begingroup$ Sorry to give you mixed messages. Since your data set is so small, I would set $k=39$ because it will give the model the most data to train with (38 records). If you had 100,000 records, I'd probably set $k=5$ depending on the type of model you are developing. Does that make sense? $\endgroup$– SkiddlesDec 5, 2018 at 15:45