3
$\begingroup$

I'm trying to experiment with the performance of different machine learning algorithms before and after applying feature selection.

I tested SVM, Random Forest, KNN, Linear Regression, and, Decision Tree and I used Random Forest as a feature selection method by utilizing the function (SelectFromModel) provided by sklearn.

All the algorithms improved significantly after applying feature selection, except the Decision Tree. I'm trying to find an explanation of why that happened? Does it mean that Decision Tree needs a large number of features to produce a good model?

Improve this question
$\endgroup$
2
  • $\begingroup$ by CART you mean a sklearn Decision tree? $\endgroup$
    – Carlos Mougan
    Commented Apr 20, 2020 at 16:04
  • $\begingroup$ @CarlosMougan Yes. Sorry for the confusion. I modified the question. $\endgroup$
    – NST
    Commented Apr 23, 2020 at 15:28

2 Answers 2

Reset to default
1
$\begingroup$

A decision tree has implicit feature selection during the model building process. That is, when it is building the tree, it only does so by splitting on features that cause the greatest increase in node purity, so features that a feature selection method would have eliminated aren’t used in the model anyway.

This is different than say a random forest where each split is chosen on only a subset of features, so it is possible that a given split is chosen from only “bad” features. By performing feature selection you remove this possibility and improve performance in the forest

Improve this answer
$\endgroup$
0
$\begingroup$

I understand CART as a Classification and Regression Tree, as just a Decision tree. Decision Trees are greedy and deterministic.

Random Forest is an ensemble of decision trees, so you will be using random trees ensembling to select features.

I am guessing, your question is not really descriptive about the situation, that since the decision tree is not pruned and uses all data it gives a good solution there is no need to take care of overfitting or pruning They normally achieve better in the train than Random Forest because they overfit easily.

Taking out features will just give the tree less information, so less power.

Does it mean that CART needs a large number of features to produce a good model?

Decision trees do good with a small number of features, it is not related.

Just to check, are you evaluating in the test set right?

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yes. I'm evaluating for the test set $\endgroup$
    – NST
    Commented Apr 23, 2020 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.