Getting lower performance metrics when using GridSearchCV

Question

I have defined an XGBoost model and would like to tune some of its hyperparameters.

I am using GridSearchCV to find the best params. However, I also tried to fit the model on the entire training dataset, and I have noticed that the 'roc_auc' performance metric is higher than when I used the Grid Search. I was surprised, because I was expecting Grid Search to perform better. I think I am missing the intuition here.

My understanding was that for grid search cross-validation, for say k folds, given a parameter value from the param_grid, gridsearchcv fits the model on the folds separately and calculates the desired performance metric. Later, for that particular parameter, it takes the 'average' of all the folds' calculated 'roc_auc'. The gridsearch repeats this process for all the other given parameters in the params_grid. Finally, the '.best_params_' is the one for which the calculated metric is higher.

This is what I tried:

param_test = {'max_depth':[3,5,6,7,9]}
model = XGBClassifier(learning_rate=0.3,
                     n_estimators=16,
                     max_depth=6,
                     min_child_weight=1,
                     gamma=0,
                     subsample=1,
                     colsample_bytree=1,
                     objective='binary:logistic',
                     nthread=4,
                     scale_pos_weight=1,
                     random_state=27)
gsearch = GridSearchCV(estimator=model, param_grid = param_test, scoring='roc_auc', cv=5)
gsearch.fit(X_train, y_train)
print('Best found params: {}'.format(gsearch.best_params_))
print('Best (Train) AUC Score: {:.4f}%'.format(gsearch.best_score_*100))

This prints:

Best found params: {'max_depth': 6}
Best (Train) AUC Score: 87.2186%

Now, when I use the same model and fit it on the entire training dataset, this is what I get:

model = XGBClassifier(learning_rate=0.3,
                     n_estimators=16,
                     max_depth=6,
                     min_child_weight=1,
                     gamma=0,
                     subsample=1,
                     colsample_bytree=1,
                     objective='binary:logistic',
                     nthread=4,
                     scale_pos_weight=1,
                     random_state=27)


model.fit(X_train, y_train, eval_metric='auc')

# Predict training set:
y_pred_train = model.predict(X_train)
y_pred_proba_train = model.predict_proba(X_train)[:,1]

# Print model report:
auc_score_train = roc_auc_score(y_train, y_pred_proba_train)

print("AUC Score (Train): {:.4f}%".format(auc_score_train*100))

which prints:

AUC Score (Train): 97.0311%

Why is there such a discrepancy? What am I missing here?

Answer 1

Your confusion seems to stem from this line:

print('Best (Train) AUC Score: {:.4f}%'.format(gsearch.best_score_*100))

The best_score_ is not exactly a training score (nor is it an unbiased estimate of future performance*): as you say, it's the averaged score across different fold splits, but each of the scores that get averaged are the performance of the models on their test fold. So, this score reflects performance of models on unseen data.

But, when you compute the score of the model retrained on the entire training set, that model has seen the training data, and the score there is inflated (quite a lot) as you'd expect.

* This is discussed at length elsewhere, but in short, while the scores are based on performance on data unseen by the models, you have looked at that data when selecting the "best" model, so to use that score now would be optimistically biased.

Answer 2

You used GridSearchCV to try max depths of [3,5,6,7,9]. It turns out that a depth of 6 gave you the best score. For your model trained on all of the data, you built it with a max depth of 6. This appears to be the same model as the best one from your grid search, only trained on more data. It makes sense that it will have better a AUC score since it is the same modeling approach but it had more examples to learn from.