Hyperparameter tuning and cross validation

Question

I have some confusion about proper usage of cross-validation to

1. tune hyperparameters and
2. evaluate estimator performance and generalizeability.

As I understand it, this would be the process you would follow:

1. Split your full dataset into a training and test set (Python's train_test_split)
2. Use cross-validation to build a model and tune hyperparameters on the training set (GridSearchCV)
3. Evaluate the best estimator and assess generalizeability using cross-validation on the test set (cross_val_score)

I've looked through sklearn's cross-validation documentation, and it recommends still having a test set for final evaluation.

A test set should still be held out for final evaluation, but the validation set is no longer needed when doing CV.

sklearn's grid-search information recommends:

When evaluating the resulting model it is important to do it on held-out samples that were not seen during the grid search process: it is recommended to split the data into a development set (to be fed to the GridSearchCV instance) and an evaluation set to compute performance metrics.

This can be done by using the train_test_split utility function.

My issue is that I often see conflicting work (for example, just cross_val_score on the entire dataset, only GridSearchCV on the entire dataset, or just a train_test_split variant), and I am hoping to understand what are best practices and clarify where I may be wrong.

Edit: This Stack Overflow answer seems to answer my question.

Answer 1

Some of the popular ways of splitting of data that the user can validate a model:

1. Train-Test (Most popular)
2. Train-Test-Validation
3. Train-Test-Development
4. Train-Test-Dev-Val

Every way has their own pros and cons. There is no one-size-fits-all approach for getting a perfect model. Choice is typically made by the developer considering following factors:

1. Size of data
2. Diversity of data
3. Computation budget
4. Efficiency
5. Necessity

But I would recommend K-fold CV is the best way to go with the basic train-test split model.

Thank you.