I have a multi-class classification problem with class imbalance. I searched for the best metric to evaluate my model. Scikit-learn has multiple ways of calculating the F1 score. I would like to understand the differences.
What do you recommending when there is a class imbalance?
F1Score is a metric to evaluate predictors performance using the formula
F1 = 2 * (precision * recall) / (precision + recall)
where
recall = TP/(TP+FN) and precision = TP/(TP+FP)
and remember:
When you have a multiclass setting, the average parameter in the f1_score function needs to be one of these:
The first one, 'weighted' calculates de F1 score for each class independently but when it adds them together uses a weight that depends on the number of true labels of each class:
$$F1_{class1}*W_1+F1_{class2}*W_2+\cdot\cdot\cdot+F1_{classN}*W_N$$
therefore favouring the majority class.
'micro' uses the global number of TP, FN, FP and calculates the F1 directly:
$$F1_{class1+class2+class3}$$
no favouring any class in particular.
Finally, 'macro' calculates the F1 separated by class but not using weights for the aggregation:
$$F1_{class1}+F1_{class2}+\cdot\cdot\cdot+F1_{classN}$$
which resuls in a bigger penalisation when your model does not perform well with the minority classes.
The one to use depends on what you want to achieve. If you are worried with class imbalance I would suggest using 'macro'. However, it might be also worthwile implementing some of the techniques available to taclke imbalance problems such as downsampling the majority class, upsampling the minority, SMOTE, etc.
Hope this helps!
