# When using Scikit Learn Grid Search, why are my train and cv scores high, but my test score is a lot lower?

I'm using scikit learn to run some models, and am very confused as to why my test score is so much lower than my cv score and my train score.

At the start, I do a 80-20 train-test split. On the train set, I run a gridsearch with 5-fold cross validation to choose hyperparameters. refit is set to true, so after picking hyperparameters the model is refit onto the whole training set, and used to predict the test set.

When I look into cv_results_ I find that my mean_train_score (what I'm interpreting to be the train score for each k-fold cross validation loop) is really high. When I look at the mean_test_score (what I'm calling cv score), it is also really high. But then when I use my external test score, the scores are really low. This is true for all models I'm using (I'm testing 10 models). The numbers can be seen in the following picture.

Note: I'm using F1 Macro Score as measure of model performance.

(LR) Logistic Regression, (QDA) Quadratic Discriminant Analysis, (NN) Nearest Neighbors, (LSVM) Linear Support Vector Machine, (RBFSVM) Radial Basis Function Support Vector Machine, (NB) Naive Bayes, (ANN) Artificial Neural Network, (RF) Random Forests, (AB) AdaBoost Random Forests, (GB) Gradient Boosted Random Forests

So since my test set performance is a lot lower than my training score, I'm sure I'm overfitting. But I don't know why my CV score would do so well then? If my setup is prone to overfitting, wouldn't I see overfitting with the 4/5 of my train set when I did the 5-fold cross validation, meaning my CV score would be low too? I don't see why I wouldn't overfit leading to high CV scores in the 5-fold CV step, but overfit for low performance in the testing set.

• How have you done the 80-20 split? – Ben Reiniger Jul 2 at 16:16
• I used the following line of code. X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state = seed, stratify = y) Thanks for the help! – tlhwu Jul 2 at 16:40
• I would try to investigate the problem by not using the 'refit' option. Run the Grid Search CV yourself, obtain the best parameters and train a new model on all of the training data after you input those best parameters. – fractalnature Jul 2 at 20:11
• Is your data 'naturally' split into groups (like different days / users / etc)? – Itamar Mushkin Jul 8 at 6:18