Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am trying to perform a classification using a decision tree classifier. I was wondering whether using a Feature reduction method is relevant for decision trees since they automatically use pruning?
My idea would be to perform a loop from 5 to 15 parameter reduction and then compare the classification accuracy of each decision tree, and then conclude the optimal number of parameters for my classification.
Thank you.
Pruning and feature reduction are different things.
Pruning: It basically compares the purity of the two leafs separately, and, together. If the leafs together are purer, than the two leafs are pruned. Thus, the decision over the parameter(s) at the node is wiped off.
Let's say you have N different parameters. You tree might be tall enough such that pruning has been used over all the parameters at different nodes. And in the same time, all these parameters might have been used in other nodes. If not, the decision tree will take the decision itself not to use this parameter - doesn't prevent from overfitting though.
Dimensionality Reduction: It you reduce the number of parameters, then these parameters would never appear in your tree, at any node. Whereas they might have been relevant at some point.
They are not uncompatible, and performing a dimensionality reduction may increase the accuracy of your task for a further classifier (as decision trees).
However, decision trees are also used for dimensionality reduction: After being trained, one might scan the features importance within the decision tree, i.e. how much each feature is used to create a split a different nodes. based on this new knowledge, you can used only the most important features to train another classifier.