I am using Repeated K-folds (RepeatedKFold(n_splits=10, n_repeats=10, random_state=999)
from sklearn) to provide reliable scores for a linear regression on my dataset.
The dataset has some outliers that should stay, and similar cases can be seen in future observations. When trained data in a fold tries to predict such observations, I get negative scores (at least, this is my interpretation).
Question: The main question is, What should I do with one (or a few) bad scores out of many? How should I report them, and how useful would that be?
Using 10 splits and 10 repeats for a dataset of size ~3000 observations, I will get 100 r-squared scores, which are all in a good range (0.97
to 0.99
). There is only one guy ruining the game, and the score is so bad (-11535
) that I cannot even get an average!
[ 9.87345591e-01 9.73912516e-01 ... -1.15353090e+04 ... 9.72986827e-01]
What should I do in this case? how to report it and/or how to cure it?
maybe it's instead that the model fit failed that one time?
@ggagliano, yes and R2 can be negative, means the model is worse than the horizontal line $\endgroup$