3
$\begingroup$

I'm doing classification using python. I'm using the class GridSearchCV, this class has the attribute best_score_ defined as "Mean cross-validated score of the best_estimator".

With this class i can also compute the score over the test set using score.

Now, I understand the theoretical difference between the two values(one is computed in the cross validation, the other is computed on the test set), but how should I interpret them? For example, if in case 1 I get these values (respectively as best_score_ and as score on the test set) 0.9236840458731027 0.8483477781024932, and in case 2 these other values 0.8923046854943018 0.8733431353820776, which one should I prefer and why? Why can the difference between the two values ​​vary so much?

$\endgroup$
  • 1
    $\begingroup$ Can you clarify your scores listed? Case 1 is CV (best score - 0.92, and score on test set - 0.84) and Case 2 is without CV (score on training data - 0.89, and score on test data- 0.87)? $\endgroup$ – fractalnature Jul 17 '20 at 17:14
  • $\begingroup$ @fractalnature i did 2 runs: in the first run i runned an algorithm with some parameters and the value of gridsearchcv.best_score_ was 0.92, in the same run the score of gridsearchcv.score(test_x, test_y) was 0.84; in the second run i runned another algorithm with some others parameters and the value of gridsearchcv.best_score_ was 0.89 and in the same run the score of gridsearchcv.score(test_x, test_y) was 0.87. $\endgroup$ – fabianod Jul 18 '20 at 20:36
  • $\begingroup$ Can you please copy paste your code for the two cases? The purpose of grid search is to do cv but using every combination of parameters you provide. So did you use a different grid? Anyway i think it would help if you included that code. $\endgroup$ – fractalnature Jul 20 '20 at 2:19
0
$\begingroup$

Gridsearch cross-validation can be used to learn the hyper-parameters of a prediction function. Consider that learning and testing of the model on the same data is a big mistake. The chance of having a perfect score but failing to predict anything useful on yet-unseen data (i.e., overfitting) is very high when using the same data for learning and testing. It is common practice when training a model to hold out a part of the data as a test set to prevent overfitting and measure the performance of the model. Also, note that the best hyper-parameters can be determined by grid search techniques and the score resulted from grid search should not be used as a criterion to measure the performance of the model. Please refer to this page for more information

That being said, best_score_ from GridSearchCV is the mean cross-validated score of the best_estimator. For example, in the case of using 5-fold cross-validation, GridSearchCV divides the data into 5 folds and trains the model 5 times. Each time, it puts one fold aside and trains the model based on the remaining 4 folds. Then, it measures the performance of the model based on the left-out fold. Finally, it returns back the mean of the performance of 5 models as the final score.

Now, let's answer this question: what does the best estimator mean? it is the estimator that was chosen by the search or the estimator which gave the highest score (or smallest loss if specified) on the left-out data. GridSearchCV's goal is to find the optimal hyperparameters. It receives a range of parameters as input and it finds the best ones based on the mean score explained above. Grid search trains different models based on different combinations of the input parameters and finally returns the best model or the best estimator. Hence, best_score_ is the mean score of the best estimator. It is notable that tuning hyperparameters with cross-Validation in the above context is one of the methods that helps you to prevent overfitting.

In your case, 0.8923046854943018 is the mean score of the best estimator. Let's call this score cross-validation score. For your case, I would go with the second case, because in that case there is no overfitting and the cross-validation and test scores are almost the same. In the first case, the cross-validation is significantly higher than the unseen test score and there is overfitting. It means that the model works very well on the train data but not on the unseen data.

$\endgroup$
  • $\begingroup$ are you sure about that? I mean, I don't want to doubt what you wrote but for me this is strange because as you said "GridSearchCV's goal is to find the optimal hyperparameters to prevent overfitting", but GridSearchCV returns the maximum cross-validation score regardless the test-set score, therefore most of the time it should cause overfitting and not avoid it. For example if i have 2 cases: case 1 (cross-validation score:0.9 test-set score 0.81) and case 2 (cv score 0.87 ts score 0.85), it returns the first one. $\endgroup$ – fabianod Jul 18 '20 at 20:46
  • $\begingroup$ Cross-validation (CV) is a powerful preventative measure against overfitting. Gridsearch CV tune the hyperparameters and selects the model having the highest score. Selection is based on the score for left-out fold, not the training score. The best estimator has the highest score for the left-out fold meaning that the difference between the scores for the unseen and training datasets for this model is less than the rest of trained models. In other words, this method selects the model with the lowest degree of overfitting. $\endgroup$ – nimar Jul 18 '20 at 23:42
0
$\begingroup$

Whenever you (or your computer) make decisions based on scores, those scores can no longer be relied upon as unbiased. So the best_score_, while it is based on scoring models on unseen-to-them data, is still an optimistically biased estimate of future performance. (An easy way to see this: if your hyperparameters have no effect except randomness, then choosing the highest-scoring one is obviously not actually better than the others, nor is that maximum value a good estimate of later performance.)

So your second option is better, having a higher (fresh) test score. Of course, if you had two choices up to this point and now you use the test score to select one, that is no longer an unbiased estimate of future performance!

All that said, usually best_score_ is reasonably close to test performance, especially if you don't have many hyperparameters to play with or if they have small impact on the modeling; your first option is a surprisingly large drop. One thing to consider is how large the test set is, and how representative it is. If your test set is too small to capture all the nuance of your population, but the training set is very large, then perhaps your test scores are more impacted by that noise and the cross-validated scores are actually more stable despite the selection bias.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.