# Cross-validation average score

I am using Repeated K-folds (RepeatedKFold(n_splits=10, n_repeats=10, random_state=999) from sklearn) to provide reliable scores for a linear regression on my dataset.

The dataset has some outliers that should stay, and similar cases can be seen in future observations. When trained data in a fold tries to predict such observations, I get negative scores (at least, this is my interpretation).

Question: The main question is, What should I do with one (or a few) bad scores out of many? How should I report them, and how useful would that be?  Using 10 splits and 10 repeats for a dataset of size ~3000 observations, I will get 100 r-squared scores, which are all in a good range (0.97 to 0.99). There is only one guy ruining the game, and the score is so bad (-11535) that I cannot even get an average!

[ 9.87345591e-01  9.73912516e-01  ... -1.15353090e+04 ...  9.72986827e-01]

What should I do in this case? how to report it and/or how to cure it?

• Why don't you just filter out the outliers? Sep 28, 2019 at 0:01
• These are performance results and they are an important part of the dataset. It's actually crucial to have them to be able to predict similar cases in future. If I train my model using the whole data to build the final model, I might be ok, but I can't ignore the result of my cross-validation, so the question is how to deal with it? Sep 28, 2019 at 0:08
• It's really just one out of the 100 scores? If it were due to one crazy outlier, then I'd expect to see such a score 10 times (for one fold from each of the 10 repeats), or maybe even more since training on a set including the outlier should hurt in a linear regression... maybe it's instead that the model fit failed that one time? Sep 28, 2019 at 0:12
• That negative value is a value assumed for R^2? Shoudn't it be between 0 and 1? Sep 28, 2019 at 0:28
• @Ben, It's actually a good point that such scores should be seen more times in the repeats, but now I removed the outliers using upper and lower limits of 3 standard deviation, and the negative scores are gone (I actually have multiple datasets of similar type and there were negative scores from 1 to 4 in the RepeatedKFold's list of 100, but also 1 in 5 if I use a simple 5-fold CV). Also, what do you actually mean by maybe it's instead that the model fit failed that one time? @ggagliano, yes and R2 can be negative, means the model is worse than the horizontal line Sep 28, 2019 at 0:35

• @BenReiniger It can happen when you calculate out-of-sample $R^2$ (depending on how you calculate $R^2$), even with an intercept. Since this question concerns cross validation, there is out-of-sample testing.