1
$\begingroup$


I am trying to perform a classification using a decision tree classifier. I was wondering whether using a Feature reduction method is relevant for decision trees since they automatically use pruning?

My idea would be to perform a loop from 5 to 15 parameter reduction and then compare the classification accuracy of each decision tree, and then conclude the optimal number of parameters for my classification.

Thank you.

$\endgroup$
1
$\begingroup$

Pruning and feature reduction are different things.

Pruning: It basically compares the purity of the two leafs separately, and, together. If the leafs together are purer, than the two leafs are pruned. Thus, the decision over the parameter(s) at the node is wiped off.

Let's say you have N different parameters. You tree might be tall enough such that pruning has been used over all the parameters at different nodes. And in the same time, all these parameters might have been used in other nodes. If not, the decision tree will take the decision itself not to use this parameter - doesn't prevent from overfitting though.

Dimensionality Reduction: It you reduce the number of parameters, then these parameters would never appear in your tree, at any node. Whereas they might have been relevant at some point.

They are not uncompatible, and performing a dimensionality reduction may increase the accuracy of your task for a further classifier (as decision trees).

However, decision trees are also used for dimensionality reduction: After being trained, one might scan the features importance within the decision tree, i.e. how much each feature is used to create a split a different nodes. based on this new knowledge, you can used only the most important features to train another classifier.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.