0
$\begingroup$

I am using Repeated K-folds (RepeatedKFold(n_splits=10, n_repeats=10, random_state=999) from sklearn) to provide reliable scores for a linear regression on my dataset.

The dataset has some outliers which should stay and also similar cases can be seen in future observations. When a trained data in a fold tries to predict such observations, I get negative scores (at least, this is my interpretation)

Question: the main question is what should I do with one (or a few) bad score(s) out of many? How should I report them and how useful would that be? Using 10 splits and 10 repeats for a data of size ~3000 observations, I will get 100 r-squared scores which are all in a good range (0.97 to 0.99). There is only one guy ruining the game and the score is so bad (-11535) that I cannot even get an average!

[ 9.87345591e-01 9.73912516e-01 ... -1.15353090e+04 ... 9.72986827e-01]

What shall I do in this case? how to report it and/or how to cure it?

$\endgroup$
  • $\begingroup$ Why don't you just filter out the outliers? $\endgroup$ – ggagliano Sep 28 '19 at 0:01
  • $\begingroup$ These are performance results and they are an important part of the dataset. It's actually crucial to have them to be able to predict similar cases in future. If I train my model using the whole data to build the final model, I might be ok, but I can't ignore the result of my cross-validation, so the question is how to deal with it? $\endgroup$ – towi_parallelism Sep 28 '19 at 0:08
  • $\begingroup$ It's really just one out of the 100 scores? If it were due to one crazy outlier, then I'd expect to see such a score 10 times (for one fold from each of the 10 repeats), or maybe even more since training on a set including the outlier should hurt in a linear regression... maybe it's instead that the model fit failed that one time? $\endgroup$ – Ben Reiniger Sep 28 '19 at 0:12
  • $\begingroup$ That negative value is a value assumed for R^2? Shoudn't it be between 0 and 1? $\endgroup$ – ggagliano Sep 28 '19 at 0:28
  • $\begingroup$ @Ben, It's actually a good point that such scores should be seen more times in the repeats, but now I removed the outliers using upper and lower limits of 3 standard deviation, and the negative scores are gone (I actually have multiple datasets of similar type and there were negative scores from 1 to 4 in the RepeatedKFold's list of 100, but also 1 in 5 if I use a simple 5-fold CV). Also, what do you actually mean by maybe it's instead that the model fit failed that one time? @ggagliano, yes and R2 can be negative, means the model is worse than the horizontal line $\endgroup$ – towi_parallelism Sep 28 '19 at 0:35
0
$\begingroup$

Your result is really a bit strange (it’s the R2, right, so the score makes no sense as R2 should be in a range between 0,1). When you do 10fold cv, each bit of your data will be used in one of the folds. So when 9/10 runs are okay, but one of the 10 scores in a 10th run is very bad, it could be coincidental clustering of your outliers in this one case.

For me this raises the question of robustness. So I would run „a lot“ of 10fold cv over my data to see if the problem occurs again (and to what extent). This is cheap in your case but gives you good arguments.

Also this gives you a good idea of how sensitive your model actually is to potential „strange“ outliers which might be part of real world problems when I understand you correctly.

If true, I would also check where outliers are particularly problematic if possible. I don‘t know the scale of your model, but having a look at predicted values vs actual y, may give you an idea.

$\endgroup$
  • $\begingroup$ thanks for the answer, a model can indeed can have a negative r2. It simply means that it performs worse than the horizontal line. it is actually coincidental that there are only 2 instances of a specific category in one my categorical predictors, I converted them to the nearest category and the problem is resolved. I wonder how linear regression performs when it tries to predict the the response for unseen categories and if there is a systematic way to detect such cases, or prevent such behaviour $\endgroup$ – towi_parallelism Sep 29 '19 at 14:14
  • $\begingroup$ re: negative R2, it can't happen in OLS with intercept fitted, but otherwise it can: stats.stackexchange.com/q/12900/232706 (My interpretation of the answer there [without looking into it further yet] is that a failed convergence in finding the least-squares solution could similarly get a negative R2.) $\endgroup$ – Ben Reiniger Sep 30 '19 at 1:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.