4
$\begingroup$

This question was always in my mind. Imagine you are doing 5-10 fold cross validation and one model gives you mean accuracy of 0.8, but with 0.2 standard deviation and the other one gives 0.7 with 0.05 standard deviation. Which one is better?

$\endgroup$
  • 1
    $\begingroup$ I found the following paper interesting: Benavoli, Alessio, et al. "Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis." The Journal of Machine Learning Research 18.1 (2017): 2653-2688. $\endgroup$ – Dave Jul 28 at 15:06
1
$\begingroup$

This is exactly a question I asked Sebastian Raschka (author of the great Python Machine Learning book), here you can find his answer saying "I also recommend the 1-standard error method, which basically means that you select the best model from k-fold based on pure performance, and then you select the simplest model that is within 1 standard error of that model".

The more extended explanation of his answer can be found on his github link: https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/10_eval3-cv/10-eval3-cv__notes.pdf

| improve this answer | |
$\endgroup$
0
$\begingroup$

First things First: 1. What is the problem statement i mean is it a regression problem or a classification problem. 2. How did you measure your accuracy, I mean MSE,MAPE,OOB,RMSE,SSE which one in regression problem or Accuracy, Precision, Recall or ROC if its a classification! Kindly clarify.

If its a classification problem! obviously, how could you measure the SD?assuming its a regression problem,

You need to answer the above things. Moreover, Every accuracy measure has its own state of the art use cases so please understand which accuracy measure you should go for else go with the model which has less error.

WHen it comes to variance, i assume higher the variance lower is the model.

| improve this answer | |
$\endgroup$
  • $\begingroup$ It's easy to measure standard deviation. You do 5-fold cross validation. In each fold, you get an accuracy score (or $F_1$ score or Brier score or MAE or whatever). You have five observations...calculate the standard deviation with the usual equation. Since the observations are dependent, the usual methods of inference (e.g. confidence intervals) become dubious, but the calculation is the calculation. $\endgroup$ – Dave Jul 28 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.