1
$\begingroup$

I'm really getting stuck with overfitting and I'm trying all I can to reduce it.

I want't to write a metric to help score models in a cv loop. I'm using 10x5 folds and still getting out of sample accuracy scores +/- 5-8% from the mean of my CV score. (i.e when i upload to kaggle)

Does the validation score's standard deviation have a correlation with overfitting / error and can this be used in my scoring? currently the stdev is around 5% that seems ALOT.

I propose that if I reduce the stdev of the validation scores of each k fold then this should decrease variance on out of sample predictions?

Does any one have any experience with this?

Any help welcome.

$\endgroup$
3
$\begingroup$

Does the validation score's standard deviation have a correlation with overfitting / error

Yes definitely: a high variance shows that the model is not stable across different training sets, which indicates a high risk of overfitting.

and can this be used in my scoring?

Using the std dev directly in the scoring itself, I'm not sure. I would consider this more like a kind of qualitative indication. However if two different models have similar performance but one with higher variance than the other, it's usually a good idea to select the latter.

currently the stdev is around 5% that seems ALOT.

It always depends on the specifics of the data, but yes I'd say that 5% is quite high.

I propose that if I reduce the stdev of the validation scores of each k fold then this should decrease variance on out of sample predictions?

I don't understand what you mean here.

In general the standard way to reduce overfitting is to increase the ratio between number of instances and number/complexity of the features:

  • Add more instances in the training set if possible (but it's rarely possible)
  • Apply feature selection to reduce the number of features
  • Simplify the most complex features; that depends on the task, but typically removing values which appear very rarely can help.
| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for this answer I $\endgroup$ – Lewis Morris Feb 14 at 4:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.