1
$\begingroup$

I have some confusion about proper usage of cross-validation to

  1. tune hyperparameters and
  2. evaluate estimator performance and generalizeability.

As I understand it, this would be the process you would follow:

  1. Split your full dataset into a training and test set (Python's train_test_split)
  2. Use cross-validation to build a model and tune hyperparameters on the training set (GridSearchCV)
  3. Evaluate the best estimator and assess generalizeability using cross-validation on the test set (cross_val_score)

I've looked through sklearn's cross-validation documentation, and it recommends still having a test set for final evaluation.

A test set should still be held out for final evaluation, but the validation set is no longer needed when doing CV.

sklearn's grid-search information recommends:

When evaluating the resulting model it is important to do it on held-out samples that were not seen during the grid search process: it is recommended to split the data into a development set (to be fed to the GridSearchCV instance) and an evaluation set to compute performance metrics.

This can be done by using the train_test_split utility function.

My issue is that I often see conflicting work (for example, just cross_val_score on the entire dataset, only GridSearchCV on the entire dataset, or just a train_test_split variant), and I am hoping to understand what are best practices and clarify where I may be wrong.

Edit: This Stack Overflow answer seems to answer my question.

$\endgroup$
1
$\begingroup$

Some of the popular ways of splitting of data that the user can validate a model:

  1. Train-Test (Most popular)
  2. Train-Test-Validation
  3. Train-Test-Development
  4. Train-Test-Dev-Val

Every way has their own pros and cons. There is no one-size-fits-all approach for getting a perfect model. Choice is typically made by the developer considering following factors:

  1. Size of data
  2. Diversity of data
  3. Computation budget
  4. Efficiency
  5. Necessity

But I would recommend K-fold CV is the best way to go with the basic train-test split model.

Thank you.

$\endgroup$
  • $\begingroup$ Nice idea: to still keep some dataset for the validation of the whole process... $\endgroup$ – TMS Oct 15 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.