294
$\begingroup$

I am trying out a multiclass classification setting with 3 classes. The class distribution is skewed with most of the data falling in 1 of the 3 classes. (class labels being 1,2,3, with 67.28% of the data falling in class label 1, 11.99% data in class 2, and remaining in class 3)

I am training a multiclass classifier on this dataset and I am getting the following performance:

                    Precision           Recall           F1-Score
Micro Average       0.731               0.731            0.731
Macro Average       0.679               0.529            0.565

I am not sure why all Micro average performances are equal and also Macro average performances are low compared to Micro average.

Improve this question
$\endgroup$
9
  • 1
    $\begingroup$ can't you look at the individual true positives etc. before averaging? also, macro averages tend to be lower than micro averages $\endgroup$
    – oW_
    Commented Dec 30, 2016 at 0:53
  • $\begingroup$ Are Micro and Macro F-measures are specific to text classification or retrieval, or they can be used for any recognition or classification problem.....If so where we can get the significance of each or any other refrence... $\endgroup$
    – idrees
    Commented Feb 19, 2018 at 7:24
  • 15
    $\begingroup$ Isn't the Micro Average Precision the same as the Accuracy of a data set? From what I understand, for Micro Average Precision you calculate the sum of all true positives and divide it by the sum of all true positives plus the sum of all false positives. So basically you divide the number of correctly identified predictions by the total number of predictions. Where is that any different from the accuracy calculation? Why do we need a new special precision term which makes things more complicated instead of simply sticking to the accuracy value? Please prove me wrong so I can sleep peacefully. $\endgroup$
    – Nico Zettler
    Commented Nov 15, 2018 at 10:06
  • 11
    $\begingroup$ @NicoZettler You are correct. Micro-averaged precision and micro-averaged recall are both equal to the accuracy when each data point is assigned to exactly one class. As to your second question, micro-averaged metrics are different from the overall accuracy when the classifications are multi-labeled (each data point may be assigned more than one label) and/or when some classes are excluded in the multi-class case. See scikit-learn.org/stable/modules/…. $\endgroup$
    – Johnson
    Commented May 7, 2019 at 17:30
  • 1
    $\begingroup$ Just to add to Nico's point, in the micro average framework there's no concept of false negative $\endgroup$
    – Tommaso Guerrini
    Commented May 13, 2020 at 11:27

8 Answers 8

Reset to default
497
$\begingroup$

Micro- and macro-averages (for whatever metric) will compute slightly different things, and thus their interpretation differs. A macro-average will compute the metric independently for each class and then take the average (hence treating all classes equally), whereas a micro-average will aggregate the contributions of all classes to compute the average metric. In a multi-class classification setup, micro-average is preferable if you suspect there might be class imbalance (i.e you may have many more examples of one class than of other classes).

To illustrate why, take for example precision $Pr=\frac{TP}{(TP+FP)}$. Let's imagine you have a One-vs-All (there is only one correct class output per example) multi-class classification system with four classes and the following numbers when tested:

  • Class A: 1 TP and 1 FP
  • Class B: 10 TP and 90 FP
  • Class C: 1 TP and 1 FP
  • Class D: 1 TP and 1 FP

You can see easily that $Pr_A = Pr_C = Pr_D = 0.5$, whereas $Pr_B=0.1$.

  • A macro-average will then compute: $Pr=\frac{0.5+0.1+0.5+0.5}{4}=0.4$
  • A micro-average will compute: $Pr=\frac{1+10+1+1}{2+100+2+2}=0.123$

These are quite different values for precision. Intuitively, in the macro-average the "good" precision (0.5) of classes A, C and D is contributing to maintain a "decent" overall precision (0.4). While this is technically true (across classes, the average precision is 0.4), it is a bit misleading, since a large number of examples are not properly classified. These examples predominantly correspond to class B, so they only contribute 1/4 towards the average in spite of constituting 94.3% of your test data. The micro-average will adequately capture this class imbalance, and bring the overall precision average down to 0.123 (more in line with the precision of the dominating class B (0.1)).

For computational reasons, it may sometimes be more convenient to compute class averages and then macro-average them. If class imbalance is known to be an issue, there are several ways around it. One is to report not only the macro-average, but also its standard deviation (for 3 or more classes). Another is to compute a weighted macro-average, in which each class contribution to the average is weighted by the relative number of examples available for it. In the above scenario, we obtain:

$Pr_{macro-mean}={0.25·0.5+0.25·0.1+0.25·0.5+0.25·0.5}=0.4$ $Pr_{macro-stdev}=0.173$

$Pr_{macro-weighted}={0.0189·0.5+0.943·0.1+0.0189·0.5+0.0189·0.5}={0.009+0.094+0.009+0.009}=0.123$

The large standard deviation (0.173) already tells us that the 0.4 average does not stem from a uniform precision among classes, but it might be just easier to compute the weighted macro-average, which in essence is another way of computing the micro-average.

Improve this answer
$\endgroup$
15
  • 48
    $\begingroup$ This answer deserves more upvotes, because it helps building an understanding why micro and macro behave differently instead of just listing the formulas (and it is original content). $\endgroup$
    – steffen
    Commented Jan 22, 2018 at 13:40
  • 6
    $\begingroup$ If you flip the scenario sketched in the reply, with the large class performing better than the small ones, you would expect to see micro average being higher than the macro average (which is the behavior reported in the question). That the macro values are different is more or less to be expected, since you are measuring different things (precision, recall...). Why the micro averages are all the same I believe is the question. $\endgroup$
    – pythiest
    Commented Feb 8, 2018 at 16:57
  • 60
    $\begingroup$ I disagree with the statement that micro average should be preferred over macro in case of imbalanced datasets. In fact, for F scores, macro is preferred over micro as the former gives equal importance to each class whereas the later gives equal importance to each sample (which means the more the number of samples, the more say it has in the final score thus favoring majority classes much like accuracy). Sources: 1. cse.iitk.ac.in/users/purushot/papers/macrof1.pdf 2. clips.uantwerpen.be/~vincent/pdf/microaverage.pdf $\endgroup$
    – shahensha
    Commented Jul 31, 2018 at 5:49
  • 8
    $\begingroup$ Is the "weighted macro-average" always going to equal the micro average? In Scikit-Learn, the definition of "weighted" is slightly different: "Calculate metrics for each label, and find their average, weighted by support (the number of true instances for each label). " From the docs for F1 Score. $\endgroup$
    – willk
    Commented Aug 6, 2018 at 20:13
  • 29
    $\begingroup$ Very good explaination, but I disagree with the part "In multi-class classification , micro-average is preferable if class is imbalance". It depends what's the objective. If it cares about overall data (not prefer to any class), 'micro' is just fine. However, let's say, the class X is rare, but it's way important, 'macro' should be a better choice because it treats each class equally. 'micro' is better if we care more about the accuracy in overall. From my view point, 'micro' is closer to 'accuracy', while 'macro' is a bit different when its not dominated by prevalent class. $\endgroup$
    – Catbuilts
    Commented Sep 20, 2019 at 8:52
46
$\begingroup$

This is the Original Post.

In Micro-average method, you sum up the individual true positives, false positives, and false negatives of the system for different sets and the apply them to get the statistics.

Tricky, but I found this very interesting. There are two methods by which you can get such average statistic of information retrieval and classification.

1. Micro-average Method

In Micro-average method, you sum up the individual true positives, false positives, and false negatives of the system for different sets and the apply them to get the statistics. For example, for a set of data, the system's

True positive (TP1)  = 12
False positive (FP1) = 9
False negative (FN1) = 3

Then precision (P1) and recall (R1) will be $57.14 \%=\frac {TP1}{TP1+FP1}$ and $80\%=\frac {TP1}{TP1+FN1}$

and for a different set of data, the system's

True positive (TP2)  = 50
False positive (FP2) = 23
False negative (FN2) = 9

Then precision (P2) and recall (R2) will be 68.49 and 84.75

Now, the average precision and recall of the system using the Micro-average method is

$\text{Micro-average of precision} = \frac{TP1+TP2}{TP1+TP2+FP1+FP2} = \frac{12+50}{12+50+9+23} = 65.96$

$\text{Micro-average of recall} = \frac{TP1+TP2}{TP1+TP2+FN1+FN2} = \frac{12+50}{12+50+3+9} = 83.78$

The Micro-average F-Score will be simply the harmonic mean of these two figures.

2. Macro-average Method

The method is straight forward. Just take the average of the precision and recall of the system on different sets. For example, the macro-average precision and recall of the system for the given example is

$\text{Macro-average precision} = \frac{P1+P2}{2} = \frac{57.14+68.49}{2} = 62.82$ $\text{Macro-average recall} = \frac{R1+R2}{2} = \frac{80+84.75}{2} = 82.25$

The Macro-average F-Score will be simply the harmonic mean of these two figures.

Suitability Macro-average method can be used when you want to know how the system performs overall across the sets of data. You should not come up with any specific decision with this average.

On the other hand, micro-average can be a useful measure when your dataset varies in size.

Improve this answer
$\endgroup$
3
  • 42
    $\begingroup$ should you give credit to this blog post? $\endgroup$
    – xiaohan2012
    Commented Aug 16, 2017 at 12:30
  • 3
    $\begingroup$ It might be worth noting that the F1-score here is not necessarily the same as the macro-averaged F1 score commonly used (like implemented in scikit or described in this paper). Usually, the F1 score is calculated for each class/set separately and then the average is calculated from the different F1 scores (here, it is done in the opposite way: first calculating the macro-averaged precision/recall and then the F1-score). $\endgroup$
    – Milania
    Commented Aug 23, 2018 at 14:55
  • $\begingroup$ FYI original link is dead $\endgroup$
    – YTZ
    Commented Jun 21, 2020 at 16:55
28
$\begingroup$

In a multi-class setting micro-averaged precision and recall are always the same.

$$ P = \frac{\sum_c TP_c}{\sum_c TP_c + \sum_c FP_c}\\ R = \frac{\sum_c TP_c}{\sum_c TP_c + \sum_c FN_c} $$ where c is the class label.

Since in a multi-class setting you count all false instances it turns out that $$ \sum_c FP_c = \sum_c FN_c $$

Hence P = R. In other words, every single False Prediction will be a False Positive for a class, and every Single Negative will be a False Negative for a class. If you treat a binary classification case as a bi-class classification and compute the micro-averaged precision and recall they will be same.

The answer given by Rahul is in the case of averaging binary precision and recall from multiple dataset. In which case the micro-averaged precision and recall are different.

Improve this answer
$\endgroup$
8
$\begingroup$

Assume that we are classifying an email into one of the three groups: urgent, normal and spam. We compare the predicts with the ground truth labels, then we get the following confusion matrix and the recall and precision for each class. enter image description here

But how can we derive a single metric that tells us how well the system is doing? There are two methods to combie these values:

enter image description here

In macroaveraging, we compute the performance for each class, and then average over classes. In microaveraging, we collect the decisions for all classes into a single confusion matrix, and then compute precision and recall from that table. The above figure shows the confusion matrix for each class separately, and shows the computation of microaveraged and macroaveraged precision.

What are the advantages and disadvantages of the two methods?

As the figure shows, a microaverage is dominated by the more frequent class (in this case spam), since the counts are pooled. The macroaverage better reflects the statistics of the smaller classes, and so is more appropriate when performance on all the classes is equally important.

In your case, since 67.28% of the data fall in class label 1, I guess that class label 1 dominates the microaverage and the performance of that class is better than other classes. If all classes are equally important the macroaverage is fairer.

Reference： Speech and Language Processing

Improve this answer
$\endgroup$
6
$\begingroup$

That's how it should be. I had the same result for my research. It seemed weird at first. But precision and recall should be the same while micro-averaging the result of multi-class single-label classifier. This is because if you consider a misclassification c1=c2 (where c1 and c2 are 2 different classes), the misclassification is a false positive (fp) with respect to c2 and false negative (fn) with respect to c1. If you sum the fn and fp for all classes, you get the same number because you are counting each misclassification as fp with respect to one class and fn with respect to another class.

Improve this answer
$\endgroup$
4
$\begingroup$

Multiclass Averaging

Introduction

I refer you to the original article for more details.

Sklearn documentation defines the average briefly:

'macro' : Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.

'micro' : Calculate metrics globally by counting the total true positives, false negatives and false positives.

Macro averaging

Macro averaging reduces your multiclass predictions down to multiple sets of binary predictions, calculates the corresponding metric for each of the binary cases, and then averages the results together. As an example, consider precision for the binary case. $P =\dfrac{TP}{TP+FP}$

In the multiclass case, macro averaging reduces the problem to multiple one-vs-all comparisons. The precision is calculated for each “relevant” column. This process is repeated for the other levels. The results are then averaged together.

The formula representation looks like this. For k classes:

$ P_{macro} = \dfrac{P_a + P_b + ... + P_n}{k}$

Note that in macro averaging, all classes get equal weight when contributing their portion of the precision value to the total. This might not be a realistic calculation when you have a large amount of class imbalance. In that case, a weighted macro average might make more sense

Micro averaging

Micro averaging treats the entire set of data as an aggregate result, and calculates 1 metric rather than k metrics that get averaged together.

For precision, this works by calculating all of the true positive results for each class and using that as the numerator, and then calculating all of the true positive and false positive results for each class, and using that as the denominator.

The formula representation looks like this. For k classes:

$ P_{micro} = \dfrac{TP_a + TP_b + ... + TP_n}{\left(TP_a + TP_b + ... + TP_n\right) + ( FP_a + FP_b + ... + FP_n)}$

In this case, rather than each class having equal weight, each observation gets equal weight. This gives the classes with the most observations more power.

Improve this answer
$\endgroup$
3
$\begingroup$

The advantage of using the Macro F1 Score is that it gives equal weight to all data points.

For example: Let's think of it as the F1 micro takes the Sum of all the Recall and Presession of different labels independently, so when we have class imbalance like:

T1 = 90% , T2 = 80% , T3=5

then F1 Micro gives equal weight to all the class and is not affected by the deviations in the distribution of the class log the Log loss it penalizes small deviations in the class.

Improve this answer
$\endgroup$
1
$\begingroup$

I think the reason why macro average is lower than micro average is well explained by pythiest's answer (dominating class has better predictions and so the micro average increase).

But the fact that micro average is equal for Precision, Recall and F1 score is because micro averaging these metrics results in overall Accuracy (as micro avg considers all classes as positive). Note that if Precision and Recall are equal then F1 score is just equal to precision/recall.

As for the question if the "weighted macro-average" always going to equal the "micro average"? I did some experiments with different no. of classes and different class imbalance and it turns out that this is not necessary true.

These statements are made with assumption that we are considering all the classes of same dataset (in contrast to Rahul Reddy Vemireddy's answer)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.