0
$\begingroup$

For context, I'm using Scikit Learn's GridSearchCV to find the best Hyperparameters of a Decision Tree.

I believe I understand Train, Validation, and Test sets and overfitting concepts when applied neural networks but those same concepts confuse me in this case.

So I split my training data in a training set and a test set.

I use the training set for GridSearchCV to do hyperparameter tuning. GridSearchCV returns a 'mean_train_score' and a 'mean_test_score'. I assume this 'mean_test_score' corresponds to the mean validation score as after choosing the best model I evalute it on the unseen test data.

So, I assume these scores:

Training score = 'mean_train_score'

Validation score = 'mean_test_score'

Test score = score on unseen test data using the fitted the model to all train data.

Is this assumption right?

I know that in neural networks when the training and validation score differ a lot from each other it may be due to overfitting and that you can apply early stopping of the traing to avoid it. So, in this case do I identify overffiting by looking at the 'mean_train_score' and 'mean_test_score' or by looking at all three scores?

What if the score on the unseen test data set is slightly higher than the 'mean_test_score'?

$\endgroup$

1 Answer 1

1
$\begingroup$

I assume this 'mean_test_score' corresponds to the mean validation score [...].

So, I assume these scores:

Training score = 'mean_train_score'

Validation score = 'mean_test_score'

Test score = score on unseen test data using the fitted the model to all train data.

That's right.

Before doing CV, you first partition the data into two sets: a train-validation set, and a test set. The test set has nothing to do with CV.

The CV object partitions the train-validation set into a train portion and a validation portion. It does this n_splits times. The scores over all training portions are averaged and recorded in mean_train_score, and the scores over all validation portions are averaged into a variable "mean_test_score".

I know that in neural networks when the training and validation score differ a lot from each other it may be due to overfitting and that you can apply early stopping of the traing to avoid it. So, in this case do I identify overffiting by looking at the 'mean_train_score' and 'mean_test_score' [...]

You can identify overfitting by comparing the CV scores on the train-validation set, i.e. mean_train_score (train folds average) and mean_test_score (validation folds average). If the train score is good, but the validation score isn't, it points to overfitting.

With sklearn algorithms, you can control overfitting by configuring models to have fewer parameters or to use their own internal early stopping if supported. You can't directly interrupt training in the same way that PyTorch/TF implement early stopping.

[...] or by looking at all three scores? What if the score on the unseen test data set is slightly higher than the 'mean_test_score'?

The test set shouldn't be used for any iterative development of the model, otherwise it ceases to be an independent measure.

The validation score is going to be an optimistic measure of performance, because you've tuned a model to optimise that score. So it's expected that the test score will be a bit worse - it's an unbiased and more realistic measure.

The test score might be higher (better) too; it's not going to be exactly the same. If it's much higher, that's likely sample variability from a relatively small test set, or that the test wasn't stratified and thus ended up with a disproportionate number of the easier cases.

$\endgroup$
2
  • $\begingroup$ Thank you for your help! Your explanation cleared my doubts $\endgroup$ Commented May 21 at 14:14
  • $\begingroup$ My pleasure. Glad it was helpful. $\endgroup$ Commented May 21 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.