5
$\begingroup$

In decision tree classifier most of the algorithms use Information gain as spiting criterion. We select the feature with maximum information gain to split on.

I think that using accuracy instead of information gain is simpler approach. Is there any scenario where accuracy doesn't work and information gain does?

Can anyone explain what are the advantages of using Information gain over accuracy as splitting criterion?

$\endgroup$
  • $\begingroup$ When you say easier, do you mean in terms of implementation? $\endgroup$ – Valentin Calomme Oct 10 '17 at 20:19
4
$\begingroup$

Decision trees are generally prone to over-fitting and accuracy doesn't generalize well to unseen data. One advantage of information gain is that -- due to the factor $-p*log(p)$ in the entropy definition -- leafs with a small number of instances are assigned less weight ($lim_{p \rightarrow 0^{+} } p*log(p) = 0$) and it favors dividing data into bigger but homogeneous groups. This approach is usually more stable and also chooses the most impactful features close to the root of the tree.

EDIT: Accuracy is usually problematic with unbalanced data. Consider this toy example:

Weather Wind    Outcome
Sunny   Weak    YES
Sunny   Weak    YES
Rainy   Weak    YES
Cloudy  Medium  YES
Rainy   Medium  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO

Weather and Wind both produce only one incorrect label hence have the same accuracy of 16/17. However, given this data, we would assume that weak winds (75% YES) are more predictive for a positive outcome than sunny weather (50% YES). That is, wind teaches us more about both outcomes. Since there are only few data points for positive outcomes we favor wind over weather, because wind is more predictive on the smaller label set which we would hope to give us a rule that is more robust to new data.

The entropy of the outcome is $ -4/17*log_2(4/17)-14/17*log_2(14/17)) =0.72$. The entropy for weather and outcome is $14/17*(-1/14*log_2(1/14)-13/14*log_2(13/14)) = 0.31$ which leads to an information gain of $0.41$. Similarly, wind gives a higher information gain of $0.6$.

$\endgroup$
  • $\begingroup$ Thank you for your answer. Could you please explain more with following points- 1. How exactly the information gain will generalize well on test data? 2. Could you please explain with an example it will be very helpful. $\endgroup$ – EngineeredBrain Oct 11 '16 at 2:26
  • $\begingroup$ It this rather easy to understand that a simpler model will generalize better on test data. Indeed, if you make less specific decisions, you're less likely to be "very" wrong. By using information gain, you ensure in a way that your tree remains rather small and balanced in terms of how many instances you have on all sibling branches. If you use accuracy, you could run into a case where you have very long branches with very few instances on them (worst case where every time you only have one instance on a branch, and all the others on the other branch) and this makes your model less general $\endgroup$ – Valentin Calomme Oct 10 '17 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.