# Impurity measures in decision trees

I have recently stepped into impurity based criteria for decision trees and I was just wondering why do we really need an impurity based criteria model such as the Gini index? What if we could simply label the entity with the majority class? What would be the effect on results?

• Please explain what you mean when you say - "simply label the entity with the majority class" Apr 22, 2020 at 16:36
• @RoshanJha the probability of one child class is more than the probability of other classes then i could simply say that parent node belongs to that child class label Apr 22, 2020 at 16:44
• – D.W.
Aug 20, 2020 at 1:01

In general, every ML model needs a function which it reduces towards a minimum value.

DecisionTree uses Gini Index Or Entropy.
These are not used to Decide to which class the Node belongs to, that is definitely decided by Majority.
At every point - Algorithm has N options(based on data and features) to split. Which one to choose.

The model tries to minimize weighted Entropy Or Gini index for the split compared to the parent.
Which indirectly imply the cleanliness of split.

Entropy is not used because it uses log which has a higher computational cost.

Check this sample code

#Gini
a = [0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
p(0) = 0.4, p(1) = 0.6
gini = 0.4*(1-0.4) + 0.6*(1-0.6) #0.40

########Split - I

#Let's split and calculate the weighted dip
a1 = [0, 0, 0, 0, 1] ; a2 = [1, 1, 1, 1, 1]
gini_a1 = 0.8*(1-0.8) + 0.2*(1-0.2) #0.32
gini_a2 = 1*(1-1) # 0

#Weighted
gini = (5/10) * gini_a1 + (5/10)*gini_a2 #0.16

########Split differently
#Let's split and calculate the weighted dip
a1 = [0, 0, 0, 0] ; a2 = [1, 1, 1, 1, 1, 1]
gini_a1 = 1*(1-1) #0 - clean split
gini_a2 = 1*(1-1) # 0

#Weighted
gini = (5/10) * gini_a1 + (5/10)*gini_a2 # 0


Obviously 2nd split is better because Gini reduction is more
Case 1 - 0.40 --> 0.16
Case 2 - 0.40 --> 0

You may think, why not simply use Accuracy to decide the split.
Check this blog

• You are suggesting that we cant use Geni index as a labeling strategy for our leaf nodes? and we always do that by majority class? Apr 22, 2020 at 21:04
• Please help me understanding more about your confusion.Apology for asking again. 1. You know how Tree splits but you are suggesting another approach? Then explain a bit more about the approach. Or 2. Your question is not about split. Then again what is it. Try explaining in steps then Zoom on the step where you have the question and mark other step as you know and agree. Apr 23, 2020 at 6:12
• No worries. My question is simple, my understanding is that geni index can be used to derive spilting criteria, we start from the root node with the lowest index and then go on. This is my understanding. Now i have been reading that geni index can also be used to assign the class to leaf and other nodes which i dont understand how. As you have already mentioned that leaf nodes class is depending on majority class. So i just wanted to compare these 2 strategies i hope its clear @roshan Apr 23, 2020 at 10:05
• Ok. Gini Index can only tell about the cleanliness of the Node i.e. More of one class and less of others. By knowing the index, I am not sure how can I decide whether it is Class-A Or Class-B. If possible please share the reference. Would be good to read/learn Apr 23, 2020 at 11:30