I am trying out a multiclass classification setting with 3 classes. The class distribution is skewed with most of the data falling in 1 of the 3 classes. (class labels being 1,2,3, with 67.28% of the data falling in class label 1, 11.99% data in class 2, and remaining in class 3)

I am training a multiclass classifier on this dataset and I am getting the following performance:

                    Precision           Recall           F1-Score
Micro Average       0.731               0.731            0.731
Macro Average       0.679               0.529            0.565

I am not sure why all Micro avg. performances are equal and also why Macro average performances are so low.

  • 1
    can't you look at the individual true positives etc. before averaging? also, macro averages tend to be lower than micro averages – oW_ Dec 30 '16 at 0:53
  • Are Micro and Macro F-measures are specific to text classification or retrieval, or they can be used for any recognition or classification problem.....If so where we can get the significance of each or any other refrence... – idrees Feb 19 at 7:24

Micro- and macro-averages (for whatever metric) will compute slightly different things, and thus their interpretation differs. A macro-average will compute the metric independently for each class and then take the average (hence treating all classes equally), whereas a micro-average will aggregate the contributions of all classes to compute the average metric. In a multi-class classification setup, micro-average is preferable if you suspect there might be class imbalance (i.e you may have many more examples of one class than of other classes).

To illustrate why, take for example precision $Pr=\frac{TP}{(TP+FP)}$. Let's imagine you have a One-vs-All (there is only one correct class output per example) multi-class classification system with four classes and the following numbers when tested:

  • Class A: 1 TP and 1 FP
  • Class B: 10 TP and 90 FP
  • Class C: 1 TP and 1 FP
  • Class D: 1 TP and 1 FP

You can see easily that $Pr_A = Pr_C = Pr_D = 0.5$, whereas $Pr_B=0.1$.

  • A macro-average will then compute: $Pr=\frac{0.5+0.1+0.5+0.5}{4}=0.4$
  • A micro-average will compute: $Pr=\frac{1+10+1+1}{2+100+2+2}=0.123$

These are quite different values for precision. Intuitively, in the macro-average the "good" precision (0.5) of classes A, C and D is contributing to maintain a "decent" overall precision (0.4). While this is technically true (across classes, the average precision is 0.4), it is a bit misleading, since a large number of examples are not properly classified. These examples predominantly correspond to class B, so they only contribute 1/4 towards the average in spite of constituting 94.3% of your test data. The micro-average will adequately capture this class imbalance, and bring the overall precision average down to 0.123 (more in line with the precision of the dominating class B (0.1)).

For computational reasons, it may sometimes be more convenient to compute class averages and then macro-average them. If class imbalance is known to be an issue, there are several ways around it. One is to report not only the macro-average, but also its standard deviation (for 3 or more classes). Another is to compute a weighted macro-average, in which each class contribution to the average is weighted by the relative number of examples available for it. In the above scenario, we obtain:

$Pr_{macro-mean}={0.25·0.5+0.25·0.1+0.25·0.5+0.25·0.5}=0.4$ $Pr_{macro-stdev}=0.173$

$Pr_{macro-weighted}={0.0189·0.5+0.943·0.1+0.0189·0.5+0.0189·0.5}={0.009+0.094+0.009+0.009}=0.123$

The large standard deviation (0.173) already tells us that the 0.4 average does not stem from a uniform precision among classes, but it might be just easier to compute the weighted macro-average, which in essence is another way of computing the micro-average.

  • 2
    This answer deserves more upvotes, because it helps building an understanding why micro and macro behave differently instead of just listing the formulas (and it is original content). – steffen Jan 22 at 13:40
  • 1
    How does this explain the different macro values in the original question? – shakedzy Feb 7 at 9:54
  • 1
    If you flip the scenario sketched in the reply, with the large class performing better than the small ones, you would expect to see micro average being higher than the macro average (which is the behavior reported in the question). That the macro values are different is more or less to be expected, since you are measuring different things (precision, recall...). Why the micro averages are all the same I believe is the question. – pythiest Feb 8 at 16:57
  • I disagree with the statement that micro average should be preferred over macro in case of imbalanced datasets. In fact, for F scores, macro is preferred over micro as the former gives equal importance to each class whereas the later gives equal importance to each sample (which means the more the number of samples, the more say it has in the final score thus favoring majority classes much like accuracy). Sources: 1. cse.iitk.ac.in/users/purushot/papers/macrof1.pdf 2. clips.uantwerpen.be/~vincent/pdf/microaverage.pdf – shahensha Jul 31 at 5:49
  • 1
    Is the "weighted macro-average" always going to equal the micro average? In Scikit-Learn, the definition of "weighted" is slightly different: "Calculate metrics for each label, and find their average, weighted by support (the number of true instances for each label). " From the docs for F1 Score. – caseWestern Aug 6 at 20:13

Original Post - http://rushdishams.blogspot.in/2011/08/micro-and-macro-average-of-precision.html


In Micro-average method, you sum up the individual true positives, false positives, and false negatives of the system for different sets and the apply them to get the statistics.

Tricky, but I found this very interesting. There are two methods by which you can get such average statistic of information retrieval and classification.

1. Micro-average Method

In Micro-average method, you sum up the individual true positives, false positives, and false negatives of the system for different sets and the apply them to get the statistics. For example, for a set of data, the system's

True positive (TP1)  = 12
False positive (FP1) = 9
False negative (FN1) = 3

Then precision (P1) and recall (R1) will be $57.14 \%=\frac {TP1}{TP1+FP1}$ and $80\%=\frac {TP1}{TP1+FN1}$

and for a different set of data, the system's

True positive (TP2)  = 50
False positive (FP2) = 23
False negative (FN2) = 9

Then precision (P2) and recall (R2) will be 68.49 and 84.75

Now, the average precision and recall of the system using the Micro-average method is

$\text{Micro-average of precision} = \frac{TP1+TP2}{TP1+TP2+FP1+FP2} = \frac{12+50}{12+50+9+23} = 65.96$

$\text{Micro-average of recall} = \frac{TP1+TP2}{TP1+TP2+FN1+FN2} = \frac{12+50}{12+50+3+9} = 83.78$

The Micro-average F-Score will be simply the harmonic mean of these two figures.

2. Macro-average Method

The method is straight forward. Just take the average of the precision and recall of the system on different sets. For example, the macro-average precision and recall of the system for the given example is

$\text{Macro-average precision} = \frac{P1+P2}{2} = \frac{57.14+68.49}{2} = 62.82$ $\text{Macro-average recall} = \frac{R1+R2}{2} = \frac{80+84.75}{2} = 82.25$

The Macro-average F-Score will be simply the harmonic mean of these two figures.

Suitability Macro-average method can be used when you want to know how the system performs overall across the sets of data. You should not come up with any specific decision with this average.

On the other hand, micro-average can be a useful measure when your dataset varies in size.

  • 10
    should you give credit to this blog post? – xiaohan2012 Aug 16 '17 at 12:30
  • 1
    Yeah @xiaohan2012, he just copypasted the answer. – Manuel G Oct 21 '17 at 18:19
  • This was my first answer on Stack overflow, I was not pretty sure how to do this. Can you suggest an edit. I will accept it. Thanks – Rahul Reddy Vemireddy Oct 22 '17 at 0:14
  • It might be worth noting that the F1-score here is not necessarily the same as the macro-averaged F1 score commonly used (like implemented in scikit or described in this paper). Usually, the F1 score is calculated for each class/set separately and then the average is calculated from the different F1 scores (here, it is done in the opposite way: first calculating the macro-averaged precision/recall and then the F1-score). – Milania Aug 23 at 14:55

In a multi-class setting micro-averaged precision and recall are always the same.

$$ P = \frac{\sum_c TP_c}{\sum_c TP_c + \sum_c FP_c}\\ R = \frac{\sum_c TP_c}{\sum_c TP_c + \sum_c FN_c} $$ where c is the class label.

Since in a multi-class setting you count all false instances it turns out that $$ \sum_c FP_c = \sum_c FN_c $$

Hence P = R. In other words, every single False Prediction will be a False Positive for a class, and every Single Negative will be a False Negative for a class. If you treat a binary classification case as a bi-class classification and compute the micro-averaged precision and recall they will be same.

The answer given by Rahul is in the case of averaging binary precision and recall from multiple dataset. In which case the micro-averaged precision and recall are different.

That's how it should be. I had the same result for my research. It seemed weird at first. But precision and recall should be the same while micro-averaging the result of multi-class single-label classifier. This is because if you consider a misclassification c1=c2 (where c1 and c2 are 2 different classes), the misclassification is a false positive (fp) with respect to c2 and false negative (fn) with respect to c1. If you sum the fn and fp for all classes, you get the same number because you are counting each misclassification as fp with respect to one class and fn with respect to another class.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.