Confusion for considering accuracy or standard deviation in selecting the best parameters

Question

I have a model with a various parameters to test.

The size of the dataset I have is not really large (~500 documents).

My issue is that when I test the parameters using 10 CV, some of them produce high accuracy value but the Standard deviation value of the folds (accuracy values of the folds) is high.

ex.

Model setup 1: acc: 0.81, STD: 0.23
Model setup 2: acc: 0.76, STD: 0.05

Setup 1 has higher accuracy but the std is high, where setup 2 has lower accuracy but with more stable results.

Thus, how can I pick the best model?

Answer 1

You are perfectly right to pay attention to the std dev across CV folds, especially with a small dataset. As you observed, different models show different values for the performance but also for the std dev, so you have to arbitrate a tradeoff between performance and stability:

• The safe option is to choose the model with lower accuracy and low variance. It might not always perform optimally but at least it won't perform too bad.
• The risky option is the high accuracy, high variance model: in average it will perform best, but you have a higher risk that it actually performs poorly.

This choice depends on the context, i.e. what the model is intended for.