Macro- or micro-average for imbalanced class problems

Question

The question of whether to use macro- or micro-averages when the data is imbalanced comes up all the time.

Some googling shows that many bloggers tend to say that micro-average is the preferred way to go, e.g.:

1. Micro-average is preferable if there is a class imbalance problem.
2. On the other hand, micro-average can be a useful measure when your dataset varies in size.

A similar question in this forum suggests a similar answer.

However, this seems quite counter-intuitive. For example if we have a data set with 90%-10% class distribution then a baseline classifier can achieve 90% mico-averaged accuracy by assigning the majority class label.

This is corroborated by books, e.g. An Introduction to Information Retrieval says (page 282) "Microaveraged results are therefore really a measure of effectiveness on the large classes in a test collection. To get a sense of effectiveness on small classes, you should compute macroaveraged results."

In the end the real decision about which measure to use should be based on the relative mis-classification costs for the classes. But a quick look at the internet seems to suggest use of micro-averaging. Is this correct or misleading?

Answer 1

The choice of a metric depends on how you rank the importance of your classes and what you value from a classifier. Let's look at your example:

For example if we have a data set with 90%-10% class distribution then a baseline classifier can achieve 90% accuracy by assigning the majority class label.

One minor correction is that this way you can achieve a 90% micro-averaged accuracy. If your goal is for your classifier simply to maximize its hits and minimize its misses, this would be the way to go.

However, if you valued the minority class the most, you should switch to a macro-averaged accuracy, where you would only get a 50% score. This metric is insensitive to the imbalance of the classes and treats them all as equal.

In many applications the latter is preferable. Imagine a classification problem aiming at diagnosing a disease that appears in 1% of the population. What good is a classifier that would always predict that the patient was healthy, even if it could achieve a 99% micro-averaged accuracy on the task?

The reason why micro-averaging is prevalent is because in most tasks, we would be interested in simply maximizing the number of correct predictions the classifier makes. In these tasks no class is more important than the others.

Answer 2

I would like to suggest another dependency.

Sometimes, predicting the large class is relatively easy. Meaning, every classifier you will try(with reasonable predicting power and one that matches your problem), will get high f1-score on the large class, but it is doing a poor job to predict the small class (f1-score). So when it is important for you to predict well the small class and predicting the big class is relatively easy, I'm suggesting to use only f1-score of the small class as main metric, or using Precision-Recall AUC(PR-AUC) as main metric.

Here is an example from my research: This is a classification report I got in one of my classifiers.

In my case, class 0 is 4 times larger than class 1. All the classifiers I played with gave me high f1-score on class 0 (above 0.9) but around 0.7 of F1-score on class 1. I'm interested in predicting well class 1, and I'm fine with suffering a small loss in predicting class 0.

So, in my case, the main difference between the classifiers was reflected on how well they perform on f1-score of class 1, hence I considered f1-score of class 1 as my main evaluation metric. My secondary metric was PR-AUC, again, on class 1 predictions (as long as my classifiers keep performing pretty well on class 0, and they all did). Optimizing these metrics better reflected my needs than the averaged versions of f1 metric.

I could consider macro avg F1, and it is a reliable metric in imbalanced case. However, that would suggest that my predicting power is 82% when I know that in my case, it's not the best metric that separate good classifiers and bad ones.

So, just be real with your data and your task in hand. Understand that an evaluation metric is a tool for you to pick the best classifier that reflects your needs, it does not have to be the one with the best numbers.

Answer 3

For imbalanced class problems:

1. Use micro-averaging to weight your metric towards the largest one.

2. Use macro-averaging to weight your metric towards the smallest one

Answer 4

F1 'micro' - the micro weighs each sample equally

     class 1 accounted for 40% of the data, F1 for this class is 0.8      class 2 accounted for 60% of the data, F1 for this class is 0.2

     0.8 x 40% + 0.2 x 60% = 0.44

F1 'macro' - the macro weighs each class equally

     class 1: the F1 result = 0.8 for class 1 F1 result = 0.2 for class 2.

We do the usual arthmetic average:      (0.8 + 0.2) / 2 = 0.5

It would be the same no matter how the samples are split between two classes. The choice depends on what you want to achieve. If you're worried about class imbalances, I'd suggest using a 'macro'.