I'm trying to use GridSearchCV from scikit-learn and look at the difference between train/test metrics.
When I do a normal test/train split with RandomForestRegressor, the metrics are comparable. Something like:
Train R2: 0.97
Test R2: 0.85
However, when I try to use the same data with GridSearchCV, the testing and training metrics seem to be completely different, the Test accuracy is a large negative number instead of being something between 0 and 1.
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import GridSearchCV
...
rf_reg = RandomForestRegressor(max_depth=10,
random_state=RANDOM,
n_estimators=100,
n_jobs=N_JOBS)
param_grid = {
"min_samples_leaf": [1, 2, 4, 10]
}
grid_cv = GridSearchCV(rf_reg, param_grid, cv=5, return_train_score=True)
grid_cv.fit(X, y)
I'm surprised that when I examine the scores for testing and training, they appear to be two different metrics. The default score for RandomForestRegressor is R2, but the results for the test sets look like they're another metric entirely.
results = pd.DataFrame(grid_cv.cv_results_)
print('Train scores:\n', results['mean_train_score'])
print('Test scores:\n', results['mean_test_score'])
Train scores:
0 0.974572
1 0.963771
2 0.936328
3 0.877382
Name: mean_train_score, dtype: float64
Test scores:
0 -5.948434
1 -5.798446
2 -6.034835
3 -6.655515
Name: mean_test_score, dtype: float64
The train scores make sense to me, they should be between 0- 1 because I'm expecting R2 error metrics, the default for a RandomForestRegressor. But why are the test scores a different metric? They should also be between 0 and 1, how is it possible to get negative numbers? This does not make sense to me.
The same thing also happens with cross_val_score, I'm expecting an R2 metric, but it returns negative numbers. Even explicitly setting the scoring method to 'r2' returns negative numbers.
from sklearn.model_selection import cross_val_score
scores = cross_val_score(rf_reg, X, y, cv=5, scoring='r2')
print(scores)
[ -5.15970579 -4.67536964 -13.17643335 -2.11630272 -4.51688508]