3
$\begingroup$

I have features extracted from a small dataset, would like to reduce the dimensions by using LDA. Also want to do a SVM classification with k-fold cross-validation.

My question is: What would be the best practice: to do LDA before the CV, or to do LDA within the CV (i.e. to each train and test fold)?

$\endgroup$
1
$\begingroup$

It depends on what you want to achieve.

If you want to visualize the results of your SVM classification, then you should do it after.

If you want to reduce noise, speed up training ... or whatever reason you want to reduce the dimensionality of your problem. You should do it before.

One idea here that can be useful, is to do LDA inside a pipeline and choosing the best hyperparameter within the CV.

In this example you can check a PCA and then a logistic regression. Your case will be similar but with LDA and SVM.

| improve this answer | |
$\endgroup$
  • $\begingroup$ The goal is to classify two-classes with the SVM. $\endgroup$ – user3885769 Jan 21 at 12:50
  • $\begingroup$ Thanks, I will check the link out. Hence, in this case, doing LDA before the SVM k-fold CV would be the best practice, right? $\endgroup$ – user3885769 Jan 21 at 14:09
  • 1
    $\begingroup$ Yes, it would be better. I am not sure If I would say best practice. For me, best practice would be doing it inside a pipeline and letting CV choosing the best hyper parameter. $\endgroup$ – Carlos Mougan Jan 21 at 14:11
1
$\begingroup$

My answer would be to perform LDA in each fold of your cross-validation.

The reason is the following. Cross-validation is used as a way to get an estimate of the performance of the model. This estimate essentially attempts to answer the following question:

How will my model perform when trained on an arbitrary set of data

If you don't use cross-validation, you always run a small risk that the model is specifically good for that particular training set. And perhaps, if your training data changes, your model won't learn as well.

This is why you want to run LDA on each fold. If you don't, you risk the fact that the impact of LDA on your SVM may become negative once your training data changes.

Of course, once you've run cross-validation and found the best hyperparameter setup, you should train your model once on all of your data.

| improve this answer | |
$\endgroup$
-1
$\begingroup$

Sometimes k-fold is not necessary to bring better results than the standard LDA. It is preferable for large data set.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.