2
$\begingroup$

I was wondering if my methodology makes sense. I am using GridSearchCV with cross-validation to train and tune model hyperparameters for a bunch of different model types (e.g. Regression Trees, Ridge, Elastic net, etc.). Before fitting the models I leave out 10% of the sample for model validation using train_test_split. (see Screenshot). I select the models with the best parameters to make predictions on the unseen validation set.

Am I missing something, as I haven't seen someone doing this when evaluating model accuracy while tuning for model parameters?

enter image description here

$\endgroup$
3
$\begingroup$

Yes it makes sense. Your "validation" set is typically called the "test" set (at least in my experience). CV is creating train/validation sets to pick the hyperparams.

The most complete process would be to then build a model one more time with the best hyperparams on the 90%, and evaluate on the 10%. This evaluation gives you a reliable estimate of the generalization error. It can be slightly lower than the error you got from the the CV process for the same parameters; the hyper-parameter fitting can itself 'overfit' a bit.

Some people skip this last step, if you don't care about estimating generalization error. Whatever it is, it's still the best you can do, according to your tuning process. The upside of skipping it is, I suppose, 10% more training data.

$\endgroup$
1
  • $\begingroup$ Thanks for your answer Sean! You are right, the validation set is used to tune for hyperparameters, while the actual test set is used to evaluate the models' prediction. As far as I conclude from the sklearn.GridSearchCV documentation, the model with the best parameters will be already re-estimated on the whole training set, as the refit option is set to true by default in GridSearchCV. $\endgroup$ – Maeaex1 Feb 9 '20 at 15:40
0
$\begingroup$

I'm wondering if this process could be even better:

  1. find the best params with GridSearchCV on the WHOLE dataset (no train_test_split)

  2. evaluate the model with the best params with k-fold Cross Validation (multiple train_test_split) calculating mean values of the metrics

$\endgroup$
1
  • $\begingroup$ Welcome to DS SE! The methods you are describing involves data leakage. You must split the data before finding any params. $\endgroup$ – Benji Albert Aug 30 '20 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.