3
$\begingroup$

I'm trying to experiment with the performance of different machine learning algorithms before and after applying feature selection.

I tested SVM, Random Forest, KNN, Linear Regression, and, Decision Tree and I used Random Forest as a feature selection method by utilizing the function (SelectFromModel) provided by sklearn.

All the algorithms improved significantly after applying feature selection, except the Decision Tree. I'm trying to find an explanation of why that happened? Does it mean that Decision Tree needs a large number of features to produce a good model?

$\endgroup$
  • $\begingroup$ by CART you mean a sklearn Decision tree? $\endgroup$ – Carlos Mougan Apr 20 at 16:04
  • $\begingroup$ @CarlosMougan Yes. Sorry for the confusion. I modified the question. $\endgroup$ – NST Apr 23 at 15:28
0
$\begingroup$

I understand CART as a Classification and Regression Tree, as just a Decision tree. Decision Trees are greedy and deterministic.

Random Forest is an ensemble of decision trees, so you will be using random trees ensembling to select features.

I am guessing, your question is not really descriptive about the situation, that since the decision tree is not pruned and uses all data it gives a good solution there is no need to take care of overfitting or pruning They normally achieve better in the train than Random Forest because they overfit easily.

Taking out features will just give the tree less information, so less power.

Does it mean that CART needs a large number of features to produce a good model?

Decision trees do good with a small number of features, it is not related.

Just to check, are you evaluating in the test set right?

| improve this answer | |
$\endgroup$
  • $\begingroup$ Yes. I'm evaluating for the test set $\endgroup$ – NST Apr 23 at 15:29
0
$\begingroup$

A decision tree has implicit feature selection during the model building process. That is, when it is building the tree, it only does so by splitting on features that cause the greatest increase in node purity, so features that a feature selection method would have eliminated aren’t used in the model anyway.

This is different than say a random forest where each split is chosen on only a subset of features, so it is possible that a given split is chosen from only “bad” features. By performing feature selection you remove this possibility and improve performance in the forest

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.