I am looking through decision trees, and I do not understand what makes each of these methods different. Could someone explain clearly what the difference between these is? Thank you.

  • $\begingroup$ Welcome to the site! Look at this post on this site, and this post on medium (explains all three with example). $\endgroup$
    – Esmailian
    Apr 4, 2019 at 11:34

1 Answer 1


As I understand it, all three want to minimize the false classified data points in your data set. (Logically, if you look for what decision trees are used)

But each of them comes from another side to this problem.

gini impurity wants "better as random"

It compares the "I label random data with random labels" against the labeling after possible split by decision tree (Wish is, that you can split the tree with better outcome than "random random random")

information gain wants small trees

It uses knowledge from information theory. It models the difference between "good" and "bad" split with criteria "simple/small trees preferred". As a result of this, it want to split the data in a way, that the daughters are "pure as possible".

For the chi-square ... I have found two things: CHAID, a (seemingly complex) decision tree technique and the chi square to prune decision trees after their building.

The chi square in general has its roots in biological statistics. It gives a characteristic number how the observed distribution conform with the null hypothesis one have about this distribution. (Biology have to act like this a lot. "I observe something, I search for an explanation, I form a hypothesis, I probe if this is statistical confirmable")

For formulas please look in Wikipedia and other sources.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.