I have recently stepped into impurity based criteria for decision trees and I was just wondering why do we really need an impurity based criteria model such as the Gini index? What if we could simply label the entity with the majority class? What would be the effect on results?


In general, every ML model needs a function which it reduces towards a minimum value.

DecisionTree uses Gini Index Or Entropy.
These are not used to Decide to which class the Node belongs to, that is definitely decided by Majority.
At every point - Algorithm has N options(based on data and features) to split. Which one to choose.

The model tries to minimize weighted Entropy Or Gini index for the split compared to the parent.
Which indirectly imply the cleanliness of split.

Entropy is not used because it uses log which has a higher computational cost.

Check this sample code

a = [0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
p(0) = 0.4, p(1) = 0.6
gini = 0.4*(1-0.4) + 0.6*(1-0.6) #0.40

########Split - I

#Let's split and calculate the weighted dip
a1 = [0, 0, 0, 0, 1] ; a2 = [1, 1, 1, 1, 1]
gini_a1 = 0.8*(1-0.8) + 0.2*(1-0.2) #0.32
gini_a2 = 1*(1-1) # 0 

gini = (5/10) * gini_a1 + (5/10)*gini_a2 #0.16

########Split differently
#Let's split and calculate the weighted dip
a1 = [0, 0, 0, 0] ; a2 = [1, 1, 1, 1, 1, 1]
gini_a1 = 1*(1-1) #0 - clean split
gini_a2 = 1*(1-1) # 0 

gini = (5/10) * gini_a1 + (5/10)*gini_a2 # 0

Obviously 2nd split is better because Gini reduction is more
Case 1 - 0.40 --> 0.16
Case 2 - 0.40 --> 0

You may think, why not simply use Accuracy to decide the split.
Check this blog

  • $\begingroup$ You are suggesting that we cant use Geni index as a labeling strategy for our leaf nodes? and we always do that by majority class? $\endgroup$ Apr 22 '20 at 21:04
  • $\begingroup$ Please help me understanding more about your confusion.Apology for asking again. 1. You know how Tree splits but you are suggesting another approach? Then explain a bit more about the approach. Or 2. Your question is not about split. Then again what is it. Try explaining in steps then Zoom on the step where you have the question and mark other step as you know and agree. $\endgroup$
    – 10xAI
    Apr 23 '20 at 6:12
  • $\begingroup$ No worries. My question is simple, my understanding is that geni index can be used to derive spilting criteria, we start from the root node with the lowest index and then go on. This is my understanding. Now i have been reading that geni index can also be used to assign the class to leaf and other nodes which i dont understand how. As you have already mentioned that leaf nodes class is depending on majority class. So i just wanted to compare these 2 strategies i hope its clear @roshan $\endgroup$ Apr 23 '20 at 10:05
  • 1
    $\begingroup$ Ok. Gini Index can only tell about the cleanliness of the Node i.e. More of one class and less of others. By knowing the index, I am not sure how can I decide whether it is Class-A Or Class-B. If possible please share the reference. Would be good to read/learn $\endgroup$
    – 10xAI
    Apr 23 '20 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.