3
$\begingroup$

I'm using scikit learn to run some models, and am very confused as to why my test score is so much lower than my cv score and my train score.

At the start, I do a 80-20 train-test split. On the train set, I run a gridsearch with 5-fold cross validation to choose hyperparameters. refit is set to true, so after picking hyperparameters the model is refit onto the whole training set, and used to predict the test set.

When I look into cv_results_ I find that my mean_train_score (what I'm interpreting to be the train score for each k-fold cross validation loop) is really high. When I look at the mean_test_score (what I'm calling cv score), it is also really high. But then when I use my external test score, the scores are really low. This is true for all models I'm using (I'm testing 10 models). The numbers can be seen in the following picture.

Note: I'm using F1 Macro Score as measure of model performance.

Table summarzing Train, CV, and Test Scores

(LR) Logistic Regression, (QDA) Quadratic Discriminant Analysis, (NN) Nearest Neighbors, (LSVM) Linear Support Vector Machine, (RBFSVM) Radial Basis Function Support Vector Machine, (NB) Naive Bayes, (ANN) Artificial Neural Network, (RF) Random Forests, (AB) AdaBoost Random Forests, (GB) Gradient Boosted Random Forests

So since my test set performance is a lot lower than my training score, I'm sure I'm overfitting. But I don't know why my CV score would do so well then? If my setup is prone to overfitting, wouldn't I see overfitting with the 4/5 of my train set when I did the 5-fold cross validation, meaning my CV score would be low too? I don't see why I wouldn't overfit leading to high CV scores in the 5-fold CV step, but overfit for low performance in the testing set.

$\endgroup$
  • $\begingroup$ How have you done the 80-20 split? $\endgroup$ – Ben Reiniger Jul 2 at 16:16
  • $\begingroup$ I used the following line of code. X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state = seed, stratify = y) Thanks for the help! $\endgroup$ – tlhwu Jul 2 at 16:40
  • $\begingroup$ I would try to investigate the problem by not using the 'refit' option. Run the Grid Search CV yourself, obtain the best parameters and train a new model on all of the training data after you input those best parameters. $\endgroup$ – fractalnature Jul 2 at 20:11
  • $\begingroup$ Is your data 'naturally' split into groups (like different days / users / etc)? $\endgroup$ – Itamar Mushkin Jul 8 at 6:18
1
$\begingroup$

Here are some ideas of things to try:

  1. I would try to investigate the problem by not using the 'refit' option. Run the Grid Search CV yourself, obtain the best parameters and train a new model on all of the training data after you input those best parameters. This is just to make sure there is nothing funky going on with SKlearn.
  2. Try using another metric for classification like AUC and see if there is any difference.
  3. Also, it might be helpful if you provide more information about your data, such as the type of outcome and the type of predictors you are using
| improve this answer | |
$\endgroup$
0
$\begingroup$

How are you saving your best weights for your models? If you have a checkpoint that is evaluating the loss/accuracy of your validation set instead of your training set, then you will end up with weights that overfit to the validation set and could do poorly against the test set. Not sure what kind of setup you have though so can you tell me how you save the weights you end up applying to your test set?

| improve this answer | |
$\endgroup$
  • $\begingroup$ I use scikit learn's GridSearchCV [scikit-learn.org/stable/modules/generated/…. After going through all the folds of the cross validation loop, it chooses the best hyperparameters, and then it retrains on the whole training set, and uses that fit for the testing set. Thanks for the help! $\endgroup$ – tlhwu Jul 2 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.