0
$\begingroup$

I have built CART model using sklearn. I'm having total 6 features in training dataset and passing all of them in fit function. I've tested both criteria Gini and entropy. But whenever I plot tree using graphviz, the tree uses only 3 features in case of Gini and 4 features in case of entropy. I've also implemented CART from scratch for cross check purpose and still, Gini uses 3 features and entropy uses 4 features.

Everything else is working fine. I'm getting the result as expected in test dataset with accuracy 97%. I just want to confirm that, is this normal? Does cart chooses best features only and ignore other features to avoid the wrong classification? Also, my dataset is quite small in size, only 220 records.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

When a feature is not that informative of your target, the algorithm can choose not to use it. This can be for two reasons:

  • All the information is already in the other features, meaning it adds nothing to include it
  • There is some regularization going on, meaning your algorithm punishes complexity if it doesn't add enough

It can also be that the regularization is very explicit, meaning there is a maximum depth set as hyperparameter. In this case the other features are more informative and useful for your task.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I think first reason fits perfect in my case. As the data is small in size. I didn't get the second point clearly as this is my first attempt at decision tree. Thanks for the answer. :) $\endgroup$
    – Scorpionk
    Feb 7, 2018 at 16:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.