Why do we need the entropy of parent node in the Information Gain. Information Gain = entropy(parent) - w * entropy(children)
We can compare the entropy of the children without the need for the parent entropy.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
It's essential; you're computing gain from the parent to the same data split in the children! Not comparing children. A good split takes a high-entropy data set (lots of several different labels) and turns it into lower-entropy data sets (some labels on one child mostly, other labels on the other).
answered Nov 25, 2018 at 14:54
-
$\begingroup$ well. I can compute children entropy with only the parent data. So we don't need parent entropy to compare children entropy. For example if parent entopry is 0.5 and children entropies are 0.3 and 0.4 I can just compare -0.3 and -0.4. Why do fomula calculate information gain using parent entropy. $\endgroup$ Commented Nov 25, 2018 at 15:38
-
$\begingroup$ Oh sure the parent entropy is constant when considering many splits. You are choosing the split with lowest child entropy regardless and that doesn't need parent entropy. However you may still need to compute information gain to, for example, see if the best split even achieves a positive gain at all. That does require knowing parent entropy. $\endgroup$ Commented Nov 26, 2018 at 4:12