Cross Validation after using train-test to decide optimal algorithm to use?

Question

I am interested in training different algorithms on a data set and observing performance metrics. Currently, my approach is to train different algorithms on train data, and then evaluate performance on a test set. Then, I use a GridSearch with cross-validation to find the optimal hyper-parameters for the best-performing model and test again using those.

I am a bit confused about cross-validation - Would this be the right approach or is there a way I could/should do cross-validation for all models?

Answer 1

The main issue is to obtain a reliable estimate of performance. It's important to take into account that both stages (the selection of the algorithm and the hyper-parameter tuning) are training stages themselves, therefore the resulting model (after selection, and only this model) should be evaluated on a fresh test set. Otherwise there's a risk of overestimating the performance, in case the selected method happens to be better by chance on the test set.

The simplest way would be to consider that this is all a single stage of parameter tuning (i.e. considering the algorithm as a parameter). In this case you should separate the data into 3 sets A,B,C:

1. All the algorithms with all their combinations of parameters are trained on A, then evaluated on B.
2. The best system is selected. Then this selected algorithm+parameters is retrained on A+B. Finally the model is evaluated on C.

Step 1 can be replaced by cross-validation. In this case there's only two sets: CV on A, then selection and evaluation on B.

Keeping the two stages separate normally requires one more level, i.e. 4 sets A,B,C,D:

1. Train all the algorithms on A, evaluate them on B.
2. Pick the best algorithm. Try all the combinations of parameters for this algorithm by training on A+B then testing on C.
3. Pick the best parameters and re-train on A+B+C, then evaluate on D.

Again CV can be used, either in step 1 or 2 or both. In this case this would be nested cross-validation, it's a bit more complex but the same principle.

Answer 2

Cross-validation should be used to find high-performing algorithm and associated hyperparameters. If you have enough data, create a grid search that searches across algorithms and respective hyperparameters. If you do not have enough data, try random search or fewer algorithms and hyperparameter options.

You should only look at the test once, after the algorithm, hyperparameters, and parameters are finalized.