1
$\begingroup$

I have heard that Random Forest and other tree based machines apply some kind of implicit feature selection.

My Question is: Does this also apply for machines like the SVM? As far as I understand is the choice of the support vectors also some kind of feature selection.

$\endgroup$
0
$\begingroup$

I assume that you are referring to the fact that training a decision tree (e.g. with the C4.5 algorithm or a variant) involves the selection of the features used as conditions in the nodes. Typically at each iteration the algorithm selects the most discriminative feature by ranking all the features using for instance their information gain.

This is indeed very close to a feature selection process:

  • ranking features by their informative power w.r.t the label is a very common way to do feature selection
  • the decision tree can end up selecting only a subset of the available features, exactly as if feature selection had been applied.

SVM works very differently and (in general) doesn't have a similar process of selecting particular features. But it's true that in a loose sense any classifier does some kind of feature selection: it measures how relevant each feature is in order to predict the correct class.

However it's important to understand that a real feature selection process consists in removing features before training, i.e. the learning algorithm doesn't get the option to use the features which have been removed. This differs a lot from the above cases because:

  • if the feature selection removes too many features, the model might not be able to perform as well as with all the original features
  • if the feature selection correctly removes redundant and/or useless information, the model is likely to perform better than if all all the features were provided. This is because the algorithm can get confused by the abundance of information, either because it's harder to find what is relevant among many options or because it doesn't have enough instances to properly assess the relevance of all the features.
$\endgroup$
  • $\begingroup$ So one would expect the ML Pipeline to perform better with feature selection. I am currently working with some AutoML tools like auto-sklearn and I try to understand the effect of not choosing a feature transformer. Does this imply that the underlying data is already reduced to the features that make sense or is this often the case in practice? $\endgroup$ – Jan Sep 22 at 0:24
  • $\begingroup$ There's no general rule, feature selection is one of the tools which can be used but whether it works better with or without it completely depends on the case. The only way to know is to experiment. Feature selection is part of the whole feature engineering process, i.e. trying to arrange the features so that the model can exploit them in an optimal way. $\endgroup$ – Erwan Sep 22 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.