I am looking through decision trees, and I do not understand what makes each of these methods different. Could someone explain clearly what the difference between these is? Thank you.
1
-
$\begingroup$ Welcome to the site! Look at this post on this site, and this post on medium (explains all three with example). $\endgroup$– EsmailianApr 4, 2019 at 11:34
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
As I understand it, all three want to minimize the false classified data points in your data set. (Logically, if you look for what decision trees are used)
But each of them comes from another side to this problem.
gini impurity wants "better as random"
It compares the "I label random data with random labels" against the labeling after possible split by decision tree (Wish is, that you can split the tree with better outcome than "random random random")
information gain wants small trees
It uses knowledge from information theory. It models the difference between "good" and "bad" split with criteria "simple/small trees preferred". As a result of this, it want to split the data in a way, that the daughters are "pure as possible".
For the chi-square ... I have found two things: CHAID, a (seemingly complex) decision tree technique and the chi square to prune decision trees after their building.
The chi square in general has its roots in biological statistics. It gives a characteristic number how the observed distribution conform with the null hypothesis one have about this distribution. (Biology have to act like this a lot. "I observe something, I search for an explanation, I form a hypothesis, I probe if this is statistical confirmable")
For formulas please look in Wikipedia and other sources.
