When using Scikit Learn Grid Search, why are my train and cv scores high, but my test score is a lot lower?

Question

I'm using scikit learn to run some models, and am very confused as to why my test score is so much lower than my cv score and my train score.

At the start, I do a 80-20 train-test split. On the train set, I run a gridsearch with 5-fold cross validation to choose hyperparameters. refit is set to true, so after picking hyperparameters the model is refit onto the whole training set, and used to predict the test set.

When I look into cv_results_ I find that my mean_train_score (what I'm interpreting to be the train score for each k-fold cross validation loop) is really high. When I look at the mean_test_score (what I'm calling cv score), it is also really high. But then when I use my external test score, the scores are really low. This is true for all models I'm using (I'm testing 10 models). The numbers can be seen in the following picture.

Note: I'm using F1 Macro Score as measure of model performance.

(LR) Logistic Regression, (QDA) Quadratic Discriminant Analysis, (NN) Nearest Neighbors, (LSVM) Linear Support Vector Machine, (RBFSVM) Radial Basis Function Support Vector Machine, (NB) Naive Bayes, (ANN) Artificial Neural Network, (RF) Random Forests, (AB) AdaBoost Random Forests, (GB) Gradient Boosted Random Forests

So since my test set performance is a lot lower than my training score, I'm sure I'm overfitting. But I don't know why my CV score would do so well then? If my setup is prone to overfitting, wouldn't I see overfitting with the 4/5 of my train set when I did the 5-fold cross validation, meaning my CV score would be low too? I don't see why I wouldn't overfit leading to high CV scores in the 5-fold CV step, but overfit for low performance in the testing set.

Answer 1

Here are some ideas of things to try:

1. I would try to investigate the problem by not using the 'refit' option. Run the Grid Search CV yourself, obtain the best parameters and train a new model on all of the training data after you input those best parameters. This is just to make sure there is nothing funky going on with SKlearn.
2. Try using another metric for classification like AUC and see if there is any difference.
3. Also, it might be helpful if you provide more information about your data, such as the type of outcome and the type of predictors you are using

Answer 2

How are you saving your best weights for your models? If you have a checkpoint that is evaluating the loss/accuracy of your validation set instead of your training set, then you will end up with weights that overfit to the validation set and could do poorly against the test set. Not sure what kind of setup you have though so can you tell me how you save the weights you end up applying to your test set?