I have some confusion about proper usage of cross-validation to
- tune hyperparameters and
- evaluate estimator performance and generalizeability.
As I understand it, this would be the process you would follow:
- Split your full dataset into a training and test set (Python's
- Use cross-validation to build a model and tune hyperparameters on the training set (
- Evaluate the best estimator and assess generalizeability using cross-validation on the test set (
I've looked through sklearn's cross-validation documentation, and it recommends still having a test set for final evaluation.
A test set should still be held out for final evaluation, but the validation set is no longer needed when doing CV.
sklearn's grid-search information recommends:
When evaluating the resulting model it is important to do it on held-out samples that were not seen during the grid search process: it is recommended to split the data into a development set (to be fed to the GridSearchCV instance) and an evaluation set to compute performance metrics.
This can be done by using the train_test_split utility function.
My issue is that I often see conflicting work (for example, just
cross_val_score on the entire dataset, only
GridSearchCV on the entire dataset, or just a
train_test_split variant), and I am hoping to understand what are best practices and clarify where I may be wrong.
Edit: This Stack Overflow answer seems to answer my question.