Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am reading the gini index definition for decision tree:
Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset.
This seems to be the same as misclassification. Is Gini index just a fancy name for misclassification? Or is there really some subtle difference? Thanks!
Is Gini index just a fancy name for misclassification?
No.
Note that Gini index definition doesn't involve predicted values, and also it involves some probabilities, which are not dependent on classifier.
Also in context of decision trees, Gini impurity corresponds to each region, and is not a single value, such as missclassification rate (technically you could also count missclassification rate per region, but then you'd also ).
See this notebook for a concrete example.
To compute misclassification rate, you should specify what the method of classification is.
Gini impurity uses a random classification with the same distribution of labels as in the set. i.e., if a set had 70 positive and 30 negative examples, each example would be randomly labeled: 70% of the time as positive and 30% of the time as negative. The misclassification rate for this classifier will be:
= Pr[Positive] * Pr[Label is Negative] + Pr[Negative] * Pr[Label is Positive]
= 0.7 * 0.3 + 0.3 * 0.7 = 0.42
We can also compute misclassification rate using a different classifier method: a majority rule. In the above example, we would always predict positive. Misclassification rate will be:
= Pr[Positive] * Pr[Label is Negative] + Pr[Negative] * Pr[Label is Positive]
= 0.7 * 0 + 0.3 * 1 = 0.3
We see that Gini impurity is one specific type of misclassification rate.
