Using standard deviation as a metric for model selection

Question

I'm really getting stuck with overfitting and I'm trying all I can to reduce it.

I want't to write a metric to help score models in a cv loop. I'm using 10x5 folds and still getting out of sample accuracy scores +/- 5-8% from the mean of my CV score. (i.e when i upload to kaggle)

Does the validation score's standard deviation have a correlation with overfitting / error and can this be used in my scoring? currently the stdev is around 5% that seems ALOT.

I propose that if I reduce the stdev of the validation scores of each k fold then this should decrease variance on out of sample predictions?

Does any one have any experience with this?

Any help welcome.

Answer 1

Does the validation score's standard deviation have a correlation with overfitting / error

Yes definitely: a high variance shows that the model is not stable across different training sets, which indicates a high risk of overfitting.

and can this be used in my scoring?

Using the std dev directly in the scoring itself, I'm not sure. I would consider this more like a kind of qualitative indication. However if two different models have similar performance but one with higher variance than the other, it's usually a good idea to select the latter.

currently the stdev is around 5% that seems ALOT.

It always depends on the specifics of the data, but yes I'd say that 5% is quite high.

I propose that if I reduce the stdev of the validation scores of each k fold then this should decrease variance on out of sample predictions?

I don't understand what you mean here.

In general the standard way to reduce overfitting is to increase the ratio between number of instances and number/complexity of the features:

• Add more instances in the training set if possible (but it's rarely possible)
• Apply feature selection to reduce the number of features
• Simplify the most complex features; that depends on the task, but typically removing values which appear very rarely can help.