2
$\begingroup$

I have a model with a various parameters to test.

The size of the dataset I have is not really large (~500 documents).

My issue is that when I test the parameters using 10 CV, some of them produce high accuracy value but the Standard deviation value of the folds (accuracy values of the folds) is high.

ex.

Model setup 1: acc: 0.81, STD: 0.23
Model setup 2: acc: 0.76, STD: 0.05

Setup 1 has higher accuracy but the std is high, where setup 2 has lower accuracy but with more stable results.

Thus, how can I pick the best model?

$\endgroup$
2
$\begingroup$

You are perfectly right to pay attention to the std dev across CV folds, especially with a small dataset. As you observed, different models show different values for the performance but also for the std dev, so you have to arbitrate a tradeoff between performance and stability:

  • The safe option is to choose the model with lower accuracy and low variance. It might not always perform optimally but at least it won't perform too bad.
  • The risky option is the high accuracy, high variance model: in average it will perform best, but you have a higher risk that it actually performs poorly.

This choice depends on the context, i.e. what the model is intended for.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanx @Erwan , shortly .. I have to pray before choosing any of the options. $\endgroup$ – Ghanem Sep 22 '19 at 21:26
  • 1
    $\begingroup$ Well praying can help ;) but usually there are external factors which can help as well: for instance if the goal is to win a ML competition then it's usually worth taking the risky option of higher performance; if the goal is to deploy the model in production in a big company where you would be fired if the performance is really bad, I'd take the safe option (actually I'd explain this to my manager and ask them to make the decision ;) ) $\endgroup$ – Erwan Sep 23 '19 at 0:54
  • $\begingroup$ you encouraged me to take the risk! .. in case of bad results I'll blame you ;) $\endgroup$ – Ghanem Sep 23 '19 at 8:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.