True positives and true negatives, F1 score: multi class classification

Question

I have 4 classes for an application of classification of animal kingdom: 1 --> invertibrates; 2 --> vertibrates; 3--> mammal; 4 ---> ambhibian. Given a mixture of images the objective is to identify mammals correctly. In the confusion matrix for this example will the TP denote the class 3 (mammal)?

Q1: Therefore, in general TP is the class category which is of utmost significance? What if all the classes are equally important, then how to denote TP and TN?

Q2: How to calculate the F1 score? should it be done separately for each class? If then there will be multiple TN's w.r.t each class !!

Can somebody please help clear these confusions? Thank you.

Answer 1

In a multiclass problem there is one score for each class, counting any other class as a negative.

For example for class 1:

• TP instances are gold standard class 1 predicted as class 1
• FN instances are gold standard class 1 predicted as class 2,3 or 4
• FP instances are gold standard class 2,3 or 4 predicted as class 1
• TN instances are gold standard class 2,3 or 4 predicted as class 2,3 or 4 (here errors don't matter as long as class 1 is not involved)

In other words, the problem is evaluated as if it was a binary classification problem for every class individually. Doing the same process for every class independently (since the status of an instance depends on the target class), one obtains a different F1-score for each class.

After that, one generally calculates either the macro F1-score or the micro F1-score (or both) in order to obtain an overall performance statistic.

Answer 2

If Given a mixture of images the objective is to identify mammals correctly, then it's a binary classification problem, mammal vs non-mammal.

The confusion matrix would be 2 by 2. Something like this, imagine fraud here is mammal.

If all the classes are equally important, then it's a multiclassification problem. The confusion matrix would be n by n.

Below is a 3 by 3 example, your case would be something similar but 4 by 4.

As to the F1 score for multiclassification, you could use classification_report() from scikit-learn.

from sklearn.metrics import classification_report
print(classification_report(new_y_test,new_y_test_pred))

The output would be something like this, again, for your case imagine there is a class 3

              precision    recall  f1-score   support

           0       0.98      0.98      0.98    563729
           1       0.47      0.39      0.42     22439
           2       0.70      0.86      0.77      3822

    accuracy                           0.96    589990
   macro avg       0.71      0.74      0.72    589990
weighted avg       0.95      0.96      0.96    589990