1
$\begingroup$

I am using Repeated K-folds (RepeatedKFold(n_splits=10, n_repeats=10, random_state=999) from sklearn) to provide reliable scores for a linear regression on my dataset.

The dataset has some outliers that should stay, and similar cases can be seen in future observations. When trained data in a fold tries to predict such observations, I get negative scores (at least, this is my interpretation).

Question: The main question is, What should I do with one (or a few) bad scores out of many? How should I report them, and how useful would that be?  Using 10 splits and 10 repeats for a dataset of size ~3000 observations, I will get 100 r-squared scores, which are all in a good range (0.97 to 0.99). There is only one guy ruining the game, and the score is so bad (-11535) that I cannot even get an average!

[ 9.87345591e-01  9.73912516e-01  ... -1.15353090e+04 ...  9.72986827e-01]

What should I do in this case? how to report it and/or how to cure it?  

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Why don't you just filter out the outliers? $\endgroup$
    – ggagliano
    Sep 28, 2019 at 0:01
  • 1
    $\begingroup$ These are performance results and they are an important part of the dataset. It's actually crucial to have them to be able to predict similar cases in future. If I train my model using the whole data to build the final model, I might be ok, but I can't ignore the result of my cross-validation, so the question is how to deal with it? $\endgroup$
    – towi_parallelism
    Sep 28, 2019 at 0:08
  • $\begingroup$ It's really just one out of the 100 scores? If it were due to one crazy outlier, then I'd expect to see such a score 10 times (for one fold from each of the 10 repeats), or maybe even more since training on a set including the outlier should hurt in a linear regression... maybe it's instead that the model fit failed that one time? $\endgroup$
    – Ben Reiniger ♦
    Sep 28, 2019 at 0:12
  • $\begingroup$ That negative value is a value assumed for R^2? Shoudn't it be between 0 and 1? $\endgroup$
    – ggagliano
    Sep 28, 2019 at 0:28
  • $\begingroup$ @Ben, It's actually a good point that such scores should be seen more times in the repeats, but now I removed the outliers using upper and lower limits of 3 standard deviation, and the negative scores are gone (I actually have multiple datasets of similar type and there were negative scores from 1 to 4 in the RepeatedKFold's list of 100, but also 1 in 5 if I use a simple 5-fold CV). Also, what do you actually mean by maybe it's instead that the model fit failed that one time? @ggagliano, yes and R2 can be negative, means the model is worse than the horizontal line $\endgroup$
    – towi_parallelism
    Sep 28, 2019 at 0:35

1 Answer 1

Reset to default
0
$\begingroup$

Your result is really a bit strange (it’s the R2, right, so the score makes no sense as R2 should be in a range between 0,1). When you do 10fold cv, each bit of your data will be used in one of the folds. So when 9/10 runs are okay, but one of the 10 scores in a 10th run is very bad, it could be coincidental clustering of your outliers in this one case.

For me this raises the question of robustness. So I would run „a lot“ of 10fold cv over my data to see if the problem occurs again (and to what extent). This is cheap in your case but gives you good arguments.

Also this gives you a good idea of how sensitive your model actually is to potential „strange“ outliers which might be part of real world problems when I understand you correctly.

If true, I would also check where outliers are particularly problematic if possible. I don‘t know the scale of your model, but having a look at predicted values vs actual y, may give you an idea.

Improve this answer
$\endgroup$
2
  • $\begingroup$ thanks for the answer, a model can indeed can have a negative r2. It simply means that it performs worse than the horizontal line. it is actually coincidental that there are only 2 instances of a specific category in one my categorical predictors, I converted them to the nearest category and the problem is resolved. I wonder how linear regression performs when it tries to predict the the response for unseen categories and if there is a systematic way to detect such cases, or prevent such behaviour $\endgroup$
    – towi_parallelism
    Sep 29, 2019 at 14:14
  • 1
    $\begingroup$ @BenReiniger It can happen when you calculate out-of-sample $R^2$ (depending on how you calculate $R^2$), even with an intercept. Since this question concerns cross validation, there is out-of-sample testing. $\endgroup$
    – Dave
    Jun 29, 2021 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.