# Cross-validation average score

I am using Repeated K-folds (RepeatedKFold(n_splits=10, n_repeats=10, random_state=999) from sklearn) to provide reliable scores for a linear regression on my dataset.

The dataset has some outliers which should stay and also similar cases can be seen in future observations. When a trained data in a fold tries to predict such observations, I get negative scores (at least, this is my interpretation)

Question: the main question is what should I do with one (or a few) bad score(s) out of many? How should I report them and how useful would that be? Using 10 splits and 10 repeats for a data of size ~3000 observations, I will get 100 r-squared scores which are all in a good range (0.97 to 0.99). There is only one guy ruining the game and the score is so bad (-11535) that I cannot even get an average!

[ 9.87345591e-01 9.73912516e-01 ... -1.15353090e+04 ... 9.72986827e-01]

What shall I do in this case? how to report it and/or how to cure it?

• Why don't you just filter out the outliers? – ggagliano Sep 28 '19 at 0:01
• These are performance results and they are an important part of the dataset. It's actually crucial to have them to be able to predict similar cases in future. If I train my model using the whole data to build the final model, I might be ok, but I can't ignore the result of my cross-validation, so the question is how to deal with it? – towi_parallelism Sep 28 '19 at 0:08
• It's really just one out of the 100 scores? If it were due to one crazy outlier, then I'd expect to see such a score 10 times (for one fold from each of the 10 repeats), or maybe even more since training on a set including the outlier should hurt in a linear regression... maybe it's instead that the model fit failed that one time? – Ben Reiniger Sep 28 '19 at 0:12
• That negative value is a value assumed for R^2? Shoudn't it be between 0 and 1? – ggagliano Sep 28 '19 at 0:28
• @Ben, It's actually a good point that such scores should be seen more times in the repeats, but now I removed the outliers using upper and lower limits of 3 standard deviation, and the negative scores are gone (I actually have multiple datasets of similar type and there were negative scores from 1 to 4 in the RepeatedKFold's list of 100, but also 1 in 5 if I use a simple 5-fold CV). Also, what do you actually mean by maybe it's instead that the model fit failed that one time? @ggagliano, yes and R2 can be negative, means the model is worse than the horizontal line – towi_parallelism Sep 28 '19 at 0:35