3
$\begingroup$

I have a multi-class classification problem with class imbalance.

I search the best metric to evaluate my model.

Sklearn has multiple way of calculating F1 score.

I would like to understand the different.

What are you recommending when there is class imbalance?

$\endgroup$
8
$\begingroup$

F1Score is a metric to evaluate predictors performance using the formula

F1 = 2 * (precision * recall) / (precision + recall)

where

recall = TP/(TP+FN) and precision = TP/(TP+FP)

and remember:

enter image description here

When you have a multiclass setting, the average parameter in the f1_score function needs to be one of these:

  • 'weighted'
  • 'micro'
  • 'macro'

The first one, 'weighted' calculates de F1 score for each class independently but when it adds them together uses a weight that depends on the number of true labels of each class:

$$F1_{class1}*W_1+F1_{class2}*W_2+\cdot\cdot\cdot+F1_{classN}*W_N$$

therefore favouring the majority class.

'micro' uses the global number of TP, FN, FP and calculates the F1 directly:

$$F1_{class1+class2+class3}$$

no favouring any class in particular.

Finally, 'macro' calculates the F1 separated by class but not using weights for the aggregation:

$$F1_{class1}+F1_{class2}+\cdot\cdot\cdot+F1_{classN}$$

which resuls in a bigger penalisation when your model does not perform well with the minority classes.

The one to use depends on what you want to achieve. If you are worried with class imbalance I would suggest using 'macro'. However, it might be also worthwile implementing some of the techniques available to taclke imbalance problems such as downsampling the majority class, upsampling the minority, SMOTE, etc.

Hope this helps!

$\endgroup$
  • $\begingroup$ I do already downsampling on the training set, should I do it also on the testset? I'm really confuse on witch dataset should I do all the technique for taclke imbalance dataset. $\endgroup$ – Fractale Nov 9 '18 at 2:35
  • $\begingroup$ You want to avoid downsampling on the test set because it will artificially bias your metrics for evaluating your model's fit, which is the point of the test set. $\endgroup$ – kevins_1 Nov 9 '18 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.