28
$\begingroup$

The problem refers to decision trees building. According to Wikipedia 'Gini coefficient' should not be confused with 'Gini impurity'. However both measures can be used when building a decision tree - these can support our choices when splitting the set of items.

1) 'Gini impurity' - it is a standard decision-tree splitting metric (see in the link above);

2) 'Gini coefficient' - each splitting can be assessed based on the AUC criterion. For each splitting scenario we can build a ROC curve and compute AUC metric. According to Wikipedia AUC=(GiniCoeff+1)/2;

Question is: are both these measures equivalent? On the one hand, I am informed that Gini coefficient should not be confused with Gini impurity. On the other hand, both these measures can be used in doing the same thing - assessing the quality of a decision tree split.

$\endgroup$
32
$\begingroup$

No, despite their names they are not equivalent or even that similar.

  • Gini impurity is a measure of misclassification, which applies in a multiclass classifier context.
  • Gini coefficient applies to binary classification and requires a classifier that can in some way rank examples according to the likelihood of being in a positive class.

Both could be applied in some cases, but they are different measures for different things. Impurity is what is commonly used in decision trees.

| improve this answer | |
$\endgroup$
6
$\begingroup$

I took an example of Data with two people A and B with wealth of unit 1 and unit 3 respectively. Gini Impurity as per Wikipedia = 1 - [ (1/4)^2 + (3/4)^2 ] = 3/8

Gini coefficient as per Wikipedia would be ratio of area between red and blue line to the total area under blue line in the following graph

enter image description here

Area under red line is 1/2 + 1 + 3/2 = 3

Total area under blue line = 4

So Gini coefficient = 3/4

Clearly the two numbers are different. I will check more cases to see if they are proportional or there is an exact relationship and edit the answer.

Edit: I checked for other combinations as well, the ratio is not constant. Below is a list of few combinations I tried. enter image description here

| improve this answer | |
$\endgroup$
  • $\begingroup$ What an explanation!! $\endgroup$ – Outlier Aug 7 '17 at 9:32
  • 1
    $\begingroup$ This has nothing to do with usage in decision trees, nor answers what impurity vs coefficient means. $\endgroup$ – Sean Owen Mar 1 at 1:37
0
$\begingroup$

I think they both represent the same concept.

In classification trees, the Gini Index is used to compute the impurity of a data partition. So Assume the data partition D consisiting of 4 classes each with equal probability. Then the Gini Index (Gini Impurity) will be: Gini(D) = 1 - (0.25^2 + 0.25^2 + 0.25^2 + 0.25^2)

In CART we perform binary splits. So The gini index will be computed as the weighted sum of the resulting partitions and we select the split with the smallest gini index.

So the use of Gini Impurity (Gini Index) is not limited to binary situations.

Another term for Gini Impurity is Gini Coefficient which is used normally as a measure of income distribution.

| improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ Gini coefficient is not Gini impurity. See the links in the question $\endgroup$ – Sean Owen Sep 10 '14 at 19:15
  • 2
    $\begingroup$ Wikipedia ist not always a reliable source of information :-) $\endgroup$ – Pasmod Turing Sep 11 '14 at 13:40
  • 2
    $\begingroup$ Sure. Go look it up somewhere else: mathworld.wolfram.com/GiniCoefficient.html What makes you think Gini coefficient = Gini impurity? $\endgroup$ – Sean Owen Sep 11 '14 at 14:03
  • $\begingroup$ Look it up: books.google.de/… $\endgroup$ – Pasmod Turing Sep 11 '14 at 15:10
  • 1
    $\begingroup$ I think we are talking about decision trees. So we are in the field of machine learning! Please read the question more carefully $\endgroup$ – Pasmod Turing Sep 19 '14 at 12:38
0
$\begingroup$

I believe they represent the same thing essentially, as the so-called:

"Gini Coefficient" mainly used in Economics, measures the inequality of a numerical variable, such as income, which we can treat as a regression problem--getting the "mean of each group.

"Gini impurity" mainly used in Decision Tree learning, measures the impurity of a categorical variable, such as colour, sex, etc. which is a classification problem -- getting the "majority" of each group.

Sounds similar right? "inequality" and "impurity" are both measures of variation, which are intuitively the same concept. The difference is "inequality" for numerical variables and "impurity" for categorical variables. And both of them can be named "Gini Index".


In Light, R. J., & Margolin, B. H. (1971). An analysis of variance for categorical data, it says that as the "mean" is an undefined concept for categorical data, Gini extends the "Gini Index" from numerical data to categorical data by using pairwise difference instead of deviation from mean. TL;DR which comes to the variation for categorical responses: $$\frac1{2n}[\sum_{i\neq j}n_in_j] = \frac{n}2 - \frac1{2n}\sum^I_{i=1}n_i^2$$ where $n_i$ is the number of responses in the $i$th category, $i = 1, \cdot\cdot\cdot, I$ which is almost the same, but $\frac{n}2$ times the "Gini Impurity" nowadays, $$1 - \sum^{I}_{i=1} {p_i}^{2}$$


By the way, you said you can use ROC as method 2 to choose split point when growing a decision tree, I can't get it. Could you elaborate that?

PS: I agreed with Pasmod Turing's answer, that Wikipedia can be modified by everyone, and the "Gini Impurity" seems like an incomplete item in the wiki.

I also saw the disputes in the comments under his answer, I must say Machine Learning is originated from statistics, and statistics is the fundamental analysis tool for scientific research, thus, many concepts are the same thing in statistics, even though they have different names in different professional areas. Gini index certainly share the same name in decision tree and economics.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Remember, the context here is decision trees. "Gini index" as used in economics (though this was not the question) is most analogous to "Gini coefficient" as used in machine learning, because it depends on pairwise comparisons. AUC may be interpreted as the probability a positive instance is deemed more likely to be positive than a negative instance, and Gini coefficient = 2 x AUC - 1. But Gini impurity is something else entirely, akin to an entropy measure. I wouldn't put stock in an answer that offers no references but asks you distrust others. $\endgroup$ – Sean Owen Mar 1 at 1:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.