Timeline for How to choose a classifier after cross-validation?
Current License: CC BY-SA 3.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 13, 2016 at 18:28 | comment | added | Armon Safai | @HimaVarsha so if we decide to take the average, how would we choose the "averaged classifier"? | |
| Sep 13, 2016 at 18:19 | comment | added | Armon Safai | @stmax but how can a single model have a mean CV score of x, as you say in your last sentence? Wouldnt the mean CV score be the average of the classifier accuracies you take from each fold? | |
| Sep 13, 2016 at 10:20 | comment | added | stmax | @ArmonSafai score can be anything you choose (accuracy, f1,...). During cross validation the classifier's performance is evaluated on each fold - if you have k folds, you get k scores. The final score is the mean (average) of the k scores. But there's also standard deviation (sd) of the k scores. So which model do you pick - the first one with 0.9 +/- 0.1, or the second one with 0.89 +/- 0.01? | |
| Sep 13, 2016 at 8:54 | comment | added | Armon Safai | @stmax can you explain how a single classifier can get a CV score of mean x, for some number? What does that even mean? | |
| Sep 13, 2016 at 8:39 | comment | added | stmax | What if one model gets a CV score of mean 0.9, sd 0.1 and another model gets mean 0.89, sd 0.01. I'd choose the second one, even though it's got a lower score than the first one. Most frameworks though just pick the model with the highest mean score, ignoring the standard deviation. | |
| Sep 13, 2016 at 8:23 | comment | added | Hima Varsha | Oh sorry, yes - the one with the better test accuracy normally is what I use. But say your data is vary varied and you get very fluctuating validation accuracies, comparing the average of accuracies can also be tried. | |
| Sep 13, 2016 at 8:06 | comment | added | Armon Safai | My question is HOW do we choose the better performing model.... | |
| Sep 13, 2016 at 8:03 | comment | added | Hima Varsha | once we have used cross-validation to select the better performing model(for instance you have 2 models-linear regression or neural network), we train that model (whether it be the linear regression or the neural network) on all the data | |
| Sep 13, 2016 at 8:00 | comment | added | Armon Safai | yea thats what i mean | |
| Sep 13, 2016 at 8:00 | comment | added | Hima Varsha | when you say test, you mean validation dataset's test right? | |
| Sep 13, 2016 at 7:54 | comment | added | Armon Safai | Sooo when we do the cross validation, we just choose the model that has the highest test accuracy? | |
| Sep 13, 2016 at 7:15 | history | answered | Hima Varsha | CC BY-SA 3.0 |