1
$\begingroup$

I'm trying to use GridSearchCV from scikit-learn and look at the difference between train/test metrics.

When I do a normal test/train split with RandomForestRegressor, the metrics are comparable. Something like:
Train R2: 0.97
Test R2: 0.85

However, when I try to use the same data with GridSearchCV, the testing and training metrics seem to be completely different, the Test accuracy is a large negative number instead of being something between 0 and 1.


from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import GridSearchCV

...

rf_reg = RandomForestRegressor(max_depth=10,
                              random_state=RANDOM,
                              n_estimators=100,
                              n_jobs=N_JOBS)

param_grid = {
              "min_samples_leaf": [1, 2, 4, 10]
             }
grid_cv = GridSearchCV(rf_reg, param_grid, cv=5, return_train_score=True)
grid_cv.fit(X, y)

I'm surprised that when I examine the scores for testing and training, they appear to be two different metrics. The default score for RandomForestRegressor is R2, but the results for the test sets look like they're another metric entirely.

results = pd.DataFrame(grid_cv.cv_results_)
print('Train scores:\n', results['mean_train_score'])
print('Test scores:\n', results['mean_test_score'])

Train scores: 
0    0.974572
1    0.963771
2    0.936328
3    0.877382
Name: mean_train_score, dtype: float64

Test scores: 
0   -5.948434
1   -5.798446
2   -6.034835
3   -6.655515
Name: mean_test_score, dtype: float64

The train scores make sense to me, they should be between 0- 1 because I'm expecting R2 error metrics, the default for a RandomForestRegressor. But why are the test scores a different metric? They should also be between 0 and 1, how is it possible to get negative numbers? This does not make sense to me.

The same thing also happens with cross_val_score, I'm expecting an R2 metric, but it returns negative numbers. Even explicitly setting the scoring method to 'r2' returns negative numbers.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(rf_reg, X, y, cv=5, scoring='r2')
print(scores)
[ -5.15970579  -4.67536964 -13.17643335  -2.11630272  -4.51688508]
$\endgroup$
2
$\begingroup$

Negative R2 values can be observed when using it in the context of model validation (where we have data that is withheld from the model) because in this context, SST $\ne$ SSE + SSR. That is, this constraint does not exist due to the data splitting. This is because in the context of model validation, the value of SST is solely calculated using the observations held in the test set only (it is just the observed variance of y in the test set only, multiplied by a factor of n-1 typically), whereas, the SSE is calculated using your trained models predictions (model is of course trained on a separate data set) and the actual values of y in your test set. Thus, it is entirely possible that SSE $>$ SST if your model is extremely poor at predicting the test set, forcing R2 = 1 - $\frac{SSE}{SST}$ to be negative.

You can basically interpret a negative R2 as your model having a very low R2 in general. This is not too surprising to see from a random forest in particular which loves to fit the training set extremely well due to how exhaustive the algorithm is (often, random forests tend to fit training sets perfectly as you have seen) but do considerably worse on held out data (though still often good enough, depending on the context. In your case, clearly not good enough).

$\endgroup$
  • $\begingroup$ Thanks, this is helpful. Any ideas why a simple test/train split does not show the dramatic difference in R2 scores? The model is trained only on the train set, but performs fairly similarly with unseen test data. $\endgroup$ – edge-case May 16 '19 at 8:59
  • $\begingroup$ Sorry, I am a bit confused. Are you saying that on a different train/test split (not the same one as what you describe in your opening post) the scores look satisfactory? Then perhaps outliers/small dataset leading to large differences in observed R2 depending on the split? What were the predictions of your model when compared to the test set? And how did you tune the model then in your opening scenario with train = 0.97 and test = 0.85? $\endgroup$ – aranglol May 16 '19 at 13:14
  • 1
    $\begingroup$ I would also maybe try increasing n estimators and also, try tuning over values of max_features and maybe set max depth to be higher. Typically, I let the trees overfit as much as possible (i.e. I do not touch max depth) and then let the bagging process reduce the variance (set n estimators to be large as possible given time constraints). $\endgroup$ – aranglol May 16 '19 at 13:23
  • $\begingroup$ Gotcha, that's helpful to set n_estimators as high as possible and let the trees overfit. If I have a dataset with only 200 observations of 1000 features, is it even meaningful to try for 1000 trees? Also, the major problem was my dataset was ordered by the target, and GridSearchCV doesn't randomize the partitions, it does them in order, I tried to explain more in the answer below. $\endgroup$ – edge-case May 16 '19 at 17:45
  • $\begingroup$ If you have only 200 observations with 1000 features, probably not too be completely honest. I see that you have solved your problem though which is awesome to hear. $\endgroup$ – aranglol May 17 '19 at 4:04
0
$\begingroup$

R2 can be negative if the model is arbitrarily worse according to the sklearn documentation

So the very negative train scores were indicative of an extremely bad performance.

Why was the test performance so colossally bad in GridSearchCV when it was decent in a simple test/train split?

The main problem is that train_test_split chooses observations randomly while GridSearchCV does not!

My problem was that the dataframe was sorted by the target variable!

The GridSearchCV and cross_val_score do not make random folds. They literally take the first 20% of observations in the dataframe as fold 1, the next 20% as fold 2, etc.

Let's say my target is a range between 1-50. If I sort my dataframe by target, then all observations are in order from 1 to 50. The first fold of the cross-validation will take (for example) only observations with a target between 1-10, save this for the test, then train the model only on targets of 20-50. This is why it was performing so badly! The model was trained on a certain range, the test set only included a target range the model had never seen before!

The solution is simple. Shuffle the original dataframe before splitting into X, y for cross-validation.

df = df.sample(frac=1, random_state=0)

This solved my problem, now the test and train scores from GridSearchCV are both between 0-1, comparable to a simple train_test_split.

Lesson learned: Always shuffle a dataframe before a cross-validation - otherwise the folds will be subject to any biases in the order of how data was collected.

$\endgroup$
  • 1
    $\begingroup$ GridSearchCV's cv parameter can be supplied with a CV splitter such as KFold (or StratifiedKFold), which allow the option to shuffle the data before splitting. This means you don't have to "manually" shuffle your dataset - important if you need to maintain the ordering for some reason. $\endgroup$ – bradS May 16 '19 at 10:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.