1
$\begingroup$

I am interested in training different algorithms on a data set and observing performance metrics. Currently, my approach is to train different algorithms on train data, and then evaluate performance on a test set. Then, I use a GridSearch with cross-validation to find the optimal hyper-parameters for the best-performing model and test again using those.

I am a bit confused about cross-validation - Would this be the right approach or is there a way I could/should do cross-validation for all models?

$\endgroup$

2 Answers 2

0
$\begingroup$

The main issue is to obtain a reliable estimate of performance. It's important to take into account that both stages (the selection of the algorithm and the hyper-parameter tuning) are training stages themselves, therefore the resulting model (after selection, and only this model) should be evaluated on a fresh test set. Otherwise there's a risk of overestimating the performance, in case the selected method happens to be better by chance on the test set.

The simplest way would be to consider that this is all a single stage of parameter tuning (i.e. considering the algorithm as a parameter). In this case you should separate the data into 3 sets A,B,C:

  1. All the algorithms with all their combinations of parameters are trained on A, then evaluated on B.
  2. The best system is selected. Then this selected algorithm+parameters is retrained on A+B. Finally the model is evaluated on C.

Step 1 can be replaced by cross-validation. In this case there's only two sets: CV on A, then selection and evaluation on B.

Keeping the two stages separate normally requires one more level, i.e. 4 sets A,B,C,D:

  1. Train all the algorithms on A, evaluate them on B.
  2. Pick the best algorithm. Try all the combinations of parameters for this algorithm by training on A+B then testing on C.
  3. Pick the best parameters and re-train on A+B+C, then evaluate on D.

Again CV can be used, either in step 1 or 2 or both. In this case this would be nested cross-validation, it's a bit more complex but the same principle.

$\endgroup$
0
$\begingroup$

Cross-validation should be used to find high-performing algorithm and associated hyperparameters. If you have enough data, create a grid search that searches across algorithms and respective hyperparameters. If you do not have enough data, try random search or fewer algorithms and hyperparameter options.

You should only look at the test once, after the algorithm, hyperparameters, and parameters are finalized.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.