34
$\begingroup$

The problem refers to decision trees building. According to Wikipedia 'Gini coefficient' should not be confused with 'Gini impurity'. However both measures can be used when building a decision tree - these can support our choices when splitting the set of items.

1) 'Gini impurity' - it is a standard decision-tree splitting metric (see in the link above);

2) 'Gini coefficient' - each splitting can be assessed based on the AUC criterion. For each splitting scenario we can build a ROC curve and compute AUC metric. According to Wikipedia AUC=(GiniCoeff+1)/2;

Question is: are both these measures equivalent? On the one hand, I am informed that Gini coefficient should not be confused with Gini impurity. On the other hand, both these measures can be used in doing the same thing - assessing the quality of a decision tree split.

$\endgroup$
1

6 Answers 6

34
$\begingroup$

No, despite their names they are not equivalent or even that similar.

  • Gini impurity is a measure of misclassification, which applies in a multiclass classifier context.
  • Gini coefficient applies to binary classification and requires a classifier that can in some way rank examples according to the likelihood of being in a positive class.

Both could be applied in some cases, but they are different measures for different things. Impurity is what is commonly used in decision trees.

$\endgroup$
2
  • $\begingroup$ "Gini coefficient applies to binary classification"? According to the links in the question, the Gini coefficient is a measure of diversity used in economics. Could you add some references to your answer? $\endgroup$ Commented Dec 16, 2020 at 0:01
  • $\begingroup$ It is used that way, but also as a metric for binary classifiers. Later on the wiki, for example, it mentions its relation to AUC: en.wikipedia.org/wiki/… $\endgroup$
    – Sean Owen
    Commented Dec 16, 2020 at 18:48
6
$\begingroup$

I took an example of Data with two people A and B with wealth of unit 1 and unit 3 respectively. Gini Impurity as per Wikipedia = 1 - [ (1/4)^2 + (3/4)^2 ] = 3/8

Gini coefficient as per Wikipedia would be ratio of area between red and blue line to the total area under blue line in the following graph

enter image description here

Area under red line is 1/2 + 1 + 3/2 = 3

Total area under blue line = 4

So Gini coefficient = 3/4

Clearly the two numbers are different. I will check more cases to see if they are proportional or there is an exact relationship and edit the answer.

Edit: I checked for other combinations as well, the ratio is not constant. Below is a list of few combinations I tried. enter image description here

$\endgroup$
2
  • $\begingroup$ What an explanation!! $\endgroup$
    – Kuber
    Commented Aug 7, 2017 at 9:32
  • 1
    $\begingroup$ This has nothing to do with usage in decision trees, nor answers what impurity vs coefficient means. $\endgroup$
    – Sean Owen
    Commented Mar 1, 2020 at 1:37
1
$\begingroup$

I believe they represent the same thing essentially, as the so-called:

"Gini Coefficient" mainly used in Economics, measures the inequality of a numerical variable, such as income, which we can treat as a regression problem--getting the "mean of each group.

"Gini impurity" mainly used in Decision Tree learning, measures the impurity of a categorical variable, such as colour, sex, etc. which is a classification problem -- getting the "majority" of each group.

Sounds similar right? "inequality" and "impurity" are both measures of variation, which are intuitively the same concept. The difference is "inequality" for numerical variables and "impurity" for categorical variables. And both of them can be named "Gini Index".


In Light, R. J., & Margolin, B. H. (1971). An analysis of variance for categorical data, it says that as the "mean" is an undefined concept for categorical data, Gini extends the "Gini Index" from numerical data to categorical data by using pairwise difference instead of deviation from mean. TL;DR which comes to the variation for categorical responses: $$\frac1{2n}[\sum_{i\neq j}n_in_j] = \frac{n}2 - \frac1{2n}\sum^I_{i=1}n_i^2$$ where $n_i$ is the number of responses in the $i$th category, $i = 1, \cdot\cdot\cdot, I$ which is almost the same, but $\frac{n}2$ times the "Gini Impurity" nowadays, $$1 - \sum^{I}_{i=1} {p_i}^{2}$$


By the way, you said you can use ROC as method 2 to choose split point when growing a decision tree, I can't get it. Could you elaborate that?

PS: I agreed with Pasmod Turing's answer, that Wikipedia can be modified by everyone, and the "Gini Impurity" seems like an incomplete item in the wiki.

I also saw the disputes in the comments under his answer, I must say Machine Learning is originated from statistics, and statistics is the fundamental analysis tool for scientific research, thus, many concepts are the same thing in statistics, even though they have different names in different professional areas. Gini index certainly share the same name in decision tree and economics.

$\endgroup$
2
  • $\begingroup$ Remember, the context here is decision trees. "Gini index" as used in economics (though this was not the question) is most analogous to "Gini coefficient" as used in machine learning, because it depends on pairwise comparisons. AUC may be interpreted as the probability a positive instance is deemed more likely to be positive than a negative instance, and Gini coefficient = 2 x AUC - 1. But Gini impurity is something else entirely, akin to an entropy measure. I wouldn't put stock in an answer that offers no references but asks you distrust others. $\endgroup$
    – Sean Owen
    Commented Mar 1, 2020 at 1:35
  • $\begingroup$ In other words, "gini inequality" is used for continuous variables? However, once a continuous variable is discretized (turned into intervals we can treat as categorical values) we use "gini impurity"? $\endgroup$ Commented Apr 14 at 13:46
0
$\begingroup$

I think they both represent the same concept.

In classification trees, the Gini Index is used to compute the impurity of a data partition. So Assume the data partition D consisiting of 4 classes each with equal probability. Then the Gini Index (Gini Impurity) will be: $Gini(D) = 1 - (0.25^2 + 0.25^2 + 0.25^2 + 0.25^2)$

In CART we perform binary splits. So The gini index will be computed as the weighted sum of the resulting partitions and we select the split with the smallest gini index.

So the use of Gini Impurity (Gini Index) is not limited to binary situations.

Another term for Gini Impurity is Gini Coefficient which is used normally as a measure of income distribution.

$\endgroup$
10
  • 4
    $\begingroup$ Gini coefficient is not Gini impurity. See the links in the question $\endgroup$
    – Sean Owen
    Commented Sep 10, 2014 at 19:15
  • 2
    $\begingroup$ Wikipedia ist not always a reliable source of information :-) $\endgroup$ Commented Sep 11, 2014 at 13:40
  • 3
    $\begingroup$ Sure. Go look it up somewhere else: mathworld.wolfram.com/GiniCoefficient.html What makes you think Gini coefficient = Gini impurity? $\endgroup$
    – Sean Owen
    Commented Sep 11, 2014 at 14:03
  • 1
    $\begingroup$ I'm sure you can find people that use coefficient and impurity interchangeably within the field of ML. Gini coefficient is not misclassification error, outside of ML. It has a meaning, whose interpretation when brought back into ML is something else. If the question is, are they the same thing, then I can't see how the answer is "yes". Search for "gini coefficient" and tell me those are all about misclassification error? $\endgroup$
    – Sean Owen
    Commented Sep 11, 2014 at 16:01
  • 1
    $\begingroup$ I think we are talking about decision trees. So we are in the field of machine learning! Please read the question more carefully $\endgroup$ Commented Sep 19, 2014 at 12:38
0
$\begingroup$

Gini impurity is a special instance of Gini coefficient:

This is Gini coefficient's definition in Wikipedia:

In economics, the Gini coefficient (/ˈdʒiːni/ JEE-nee), also known as the Gini index or Gini ratio, is a measure of statistical dispersion intended to represent the income inequality or the wealth inequality within a nation or a social group.

In another words, it measures the inequality of the wealth of each person in a nation, with the constraint that the sum of their wealth is a constant.

Now replace the above bolded words with:

person -> category
wealth -> probability
nation -> probability distribution
constant -> 1

The above sentence become: it measures the inequality of the probability of each category in a probability distribution , with the constraint that the sum of their probability is 1.

That's exactly the definition of Gini impurity!

$\endgroup$
0
$\begingroup$

Gini index of 1 would represent wealth concentration to a single person. However, the Gini impurity in this case would be 0. So, they should move in opposite directions, right?

$\endgroup$
1
  • $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Mar 26, 2023 at 8:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.