3
$\begingroup$

When I have the data set to train a model with SVM, which procedure is performed first, cross validation or grid search? I have read this in a couple of books but I don't know in what order all this should be done. If cross-validation is first performed, what hyperparameters do I use there if I have not found the optimal values provided by the grid search? In addition, throughout this procedure, where should the confusion matrix be calculated?

Thanks in advance

$\endgroup$
3
$\begingroup$

Well, grid search involves finding best hyperparameters. Best according to what data set? a held out validation set. If that's what you mean by cross validation, then they necessarily happen simultaneously.

It doesn't really make sense to do something called cross validation before testing hyperparams - indeed, what would you be evaluating?

CV as in k-fold cross validation can also happen within each model fitting process in the search, to produce a better estimate of the loss (and its variance, which is useful in more sophisticated tuning procedures). I think this is less usual but valid.

It's possible to use CV when fitting the final model after hyperparameter search. It might give you a better estimate of the loss, or confusion matrix, as you compute many of them. But each model you fit isn't using all available data. I think it's probably more conventional to take the best model's parameters and loss / confusion matrix, from the fitting process, as an estimate of generalization, and then refit the final model on all data. This means no CV at that stage.

| improve this answer | |
$\endgroup$
2
$\begingroup$

Both are done together!

Grid search is how you determine the hyperparameters of a model whereas, Cross-Validation is the process of running your model on a data separate from training datase to gauge your models performance on a particular hyperparameter.

Grid search allows you to have some potential hyperparameters among which you compare the model's performance through training on the training dataset and evaluating on validation dataset.

The metric you use depends on the final objective. You can choose the hyperparameters which has the minimum loss or maximum accuracy or maximum AUROC or maximum f1-score.

The confusion matrix that you mention can be computed during training as well as during validation. It just allows you to gauge the performance of your current model. The matrix will give you 4 numbers which is combined in different ways to obtain different metrics such as: accuracy, auroc and f1-score. Among these metrics which one to use depends on the final task you want to perform.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for your response, it helped me understand the procedure in question. $\endgroup$ – SRG Mar 2 at 15:38
  • $\begingroup$ @SRG Please upvote my answer if you found it helpful :) $\endgroup$ – user1825567 Mar 2 at 20:32
  • $\begingroup$ The other answer helped me a little more for some details than he mentioned but I think that both answers are excellent, I also like to thank all the users who answer me a question for providing their help and their valuable time. $\endgroup$ – SRG Mar 3 at 21:06
  • $\begingroup$ Your welcome. I think you misunderstood 'upvoting' as 'accepting' an answer. One can 'accept' only 1 answer but can 'upvote' as many answer as possible :) $\endgroup$ – user1825567 Mar 4 at 12:20
  • $\begingroup$ Ok I understand, I'm sorry I don't speak much English. Already vote your answer. Thank you very much for explaining. $\endgroup$ – SRG Mar 5 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.