Cross validation and train_test_split

Question

I am building a class that follows the workflow:

1. Model Selection and Fitting The class accepts a list of models and their respective hyperparameter grids. It then performs a standard fitting process for each model using the provided dataset. The primary evaluation metric (scoring) can be specified by the developer (e.g., accuracy, ROC AUC, precision, mean absolute error).

2. Top Model Selection Following the model fitting stage, the class ranks the models based on their scores and selects the top N models, where N is determined by the developer. This ensures that the subsequent hyperparameter tuning is focused on the most promising models.

3. Hyperparameter Optimization For the selected top models, the class applies grid search to explore various hyperparameter combinations. The best hyperparameters are chosen for each model, and the corresponding scores are recorded. You may select grid, random or bayesian as search methods

4. Results and Rankings The class provides insights into the model rankings based on their scores post hyperparameter optimization. This ranking assists developers in identifying the most suitable models for their specific problem.

However, I'm encountering a challenge: I'm performing the initial fit using the train_test_split method (as using cross-validation could be resource-intensive). During the hyperparameter search, I'm using cross-validation since it's a standard practice (I haven't yet found a viable solution to perform a search without cross-validation).

Do you believe this is a sound approach, or could comparing train-test results with cross-validation be conceptually flawed? Would you suggest segregating these processes (comparing cross-validation with cross-validation and train-test with train-test), or do you recommend sticking to the current approach?

Answer 1

Ideally you want to use your test-set only at the very end, i.e. after hyper-parameter optimization when you know what the best model is. Otherwise you would "reuse" the test-set multiple times, risking to indirectly adapt to it: which is unwanted.

I think you should always cross-validate, or if you want to save up computation evaluate only on the train set. But in the latter case you wouldn't have an estimate of the generalization error of the model, which is something that is comparable with the test error.

CV should be the best choice, and I think you can let the user/developer decide the validation method (e.g. leave-one-out, which is cheap, or k-fold cross val) and related parameter (like $\%$ of the split, or the number of folds) according to the available computational budget.