I'm using scikit learn to run some models, and am very confused as to why my test score is so much lower than my cv score and my train score.
At the start, I do a 80-20 train-test split. On the train set, I run a gridsearch with 5-fold cross validation to choose hyperparameters. refit is set to true, so after picking hyperparameters the model is refit onto the whole training set, and used to predict the test set.
When I look into cv_results_ I find that my mean_train_score (what I'm interpreting to be the train score for each k-fold cross validation loop) is really high. When I look at the mean_test_score (what I'm calling cv score), it is also really high. But then when I use my external test score, the scores are really low. This is true for all models I'm using (I'm testing 10 models). The numbers can be seen in the following picture.
Note: I'm using F1 Macro Score as measure of model performance.
(LR) Logistic Regression, (QDA) Quadratic Discriminant Analysis, (NN) Nearest Neighbors, (LSVM) Linear Support Vector Machine, (RBFSVM) Radial Basis Function Support Vector Machine, (NB) Naive Bayes, (ANN) Artificial Neural Network, (RF) Random Forests, (AB) AdaBoost Random Forests, (GB) Gradient Boosted Random Forests
So since my test set performance is a lot lower than my training score, I'm sure I'm overfitting. But I don't know why my CV score would do so well then? If my setup is prone to overfitting, wouldn't I see overfitting with the 4/5 of my train set when I did the 5-fold cross validation, meaning my CV score would be low too? I don't see why I wouldn't overfit leading to high CV scores in the 5-fold CV step, but overfit for low performance in the testing set.