1
$\begingroup$

I have heard that Random Forest and other tree based machines apply some kind of implicit feature selection.

My Question is: Does this also apply for machines like the SVM? As far as I understand is the choice of the support vectors also some kind of feature selection.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

I assume that you are referring to the fact that training a decision tree (e.g. with the C4.5 algorithm or a variant) involves the selection of the features used as conditions in the nodes. Typically at each iteration the algorithm selects the most discriminative feature by ranking all the features using for instance their information gain.

This is indeed very close to a feature selection process:

  • ranking features by their informative power w.r.t the label is a very common way to do feature selection
  • the decision tree can end up selecting only a subset of the available features, exactly as if feature selection had been applied.

SVM works very differently and (in general) doesn't have a similar process of selecting particular features. But it's true that in a loose sense any classifier does some kind of feature selection: it measures how relevant each feature is in order to predict the correct class.

However it's important to understand that a real feature selection process consists in removing features before training, i.e. the learning algorithm doesn't get the option to use the features which have been removed. This differs a lot from the above cases because:

  • if the feature selection removes too many features, the model might not be able to perform as well as with all the original features
  • if the feature selection correctly removes redundant and/or useless information, the model is likely to perform better than if all all the features were provided. This is because the algorithm can get confused by the abundance of information, either because it's harder to find what is relevant among many options or because it doesn't have enough instances to properly assess the relevance of all the features.
Improve this answer
$\endgroup$
2
  • $\begingroup$ So one would expect the ML Pipeline to perform better with feature selection. I am currently working with some AutoML tools like auto-sklearn and I try to understand the effect of not choosing a feature transformer. Does this imply that the underlying data is already reduced to the features that make sense or is this often the case in practice? $\endgroup$
    – Jan
    Sep 22, 2019 at 0:24
  • 1
    $\begingroup$ There's no general rule, feature selection is one of the tools which can be used but whether it works better with or without it completely depends on the case. The only way to know is to experiment. Feature selection is part of the whole feature engineering process, i.e. trying to arrange the features so that the model can exploit them in an optimal way. $\endgroup$
    – Erwan
    Sep 22, 2019 at 10:34
0
$\begingroup$

Note that there are works to optimally combine feature selection and getting SVM weights.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.