I'm really getting stuck with overfitting and I'm trying all I can to reduce it.
I want't to write a metric to help score models in a cv loop. I'm using 10x5 folds and still getting out of sample accuracy scores +/- 5-8% from the mean of my CV score. (i.e when i upload to kaggle)
Does the validation score's standard deviation have a correlation with overfitting / error and can this be used in my scoring? currently the stdev is around 5% that seems ALOT.
I propose that if I reduce the stdev of the validation scores of each k fold then this should decrease variance on out of sample predictions?
Does any one have any experience with this?
Any help welcome.