9
$\begingroup$

Background: Currently I'm working on my thesis project, which is to build Tree-based ensemble methods for classification on a large data set. Before I started with modeling, I've spent a large amount of time on feature selection using correlation-based criteria to select a subset of features, so that those chosen features have a high correlation with the response variable and low correlation with each other.

However, one of my supervisors questioned why I spent so much time in feature selection, as he mentioned, the decision tree algorithm can naturally select which features are most important. Later I checked the book Introduction to Data Mining by Tan (2014) and it says clearly that "Feature selection occurs naturally as part of the data mining algorithm, ..., such as decision tree classifiers". I felt a bit frustrated by spending time on feature selection. When I eventually run the models with and without feature selection, it does NOT show any significant differences between the results.

My question is: Does feature selections matter at all to Decision Tree algorithms?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Yes, it does. By removing variables that are used not that often (lower importance) you can gain generalization (even for RFs). But, I would not select them based on the correlation with the response variable because you will be missing interactions among them. If the dataset is not large I would recommend the use of a genetic algorithm and Cross-Validation (real CV, not the internal one). If the dataset is large I would try external methods (e.g. Relieff) and not necessarily the internal variable importance as it may be biased (See Strobel papers). $\endgroup$
    – Rafael Muñoz-Mas
    May 14, 2019 at 7:47
  • $\begingroup$ I would also like to add that feature selection will help if they are lot of features to consider. $\endgroup$
    – XristosK
    Mar 30 at 7:38

2 Answers 2

Reset to default
10
$\begingroup$

For ensembles of decision trees, feature selection is generally not that important. During the induction of decision trees, the optimal feature is selected to split the data based on metrics like information gain, so if you have some non-informative features, they simply won't be selected.

Feature selection can still be important for small datasets, where spurious relationships between features and class labels are more commonly seen.

As a side note: it's common practice to remove some features from datasets for all learning algorithms, like ID fields. These features are especially bad for decision trees, as they usually end up being the feature with the highest information gain (and therefore selected to be split on) while actually containing no generalisable information. So in this way, feature selection can be useful when using decision trees, although it's arguable that the ID field should be considered a feature in the first place.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Constant variables such as ID fields may encode fixed effects which is important. $\endgroup$
    – user46827
    Mar 26 at 19:43
0
$\begingroup$

There's another benefit of feature selection in your case. You mentioned that your project is applied to large datasets. In practice, by doing feature selection before fitting a model you can speed up the fitting procedure because there are fewer data to be fed to the model.

This is especially useful when your data is wide, i.e. has many columns.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.