Evaluation methods for multi-class classification

Question

I am looking for single-number evaluation method that can be used in multi-class classification tasks that take into account imbalanced data-sets. For instance, ROC-AUC defined by binary classifiers, is a single-number and takes into account imbalanced data-sets. On the other hand, accuracy is single-number, defined for multi-class classifiers and does not take into account imbalanced data-sets. Finally, the confusion matrix is defined for multi-class, takes that into account but is not single-number. Is there any evaluation method that satisfies the three conditions?

Answer 1

How about a weighted log-loss?

Lets say we have $m$ classes $c_1, \dots, c_m$. We can give each class $c_i$ a weight $w_i$ which is inversely proportional to the percentage of the dataset that belongs to $c_i$. Then, the loss for some data set with actual classes $y = y_1, \dots, y_n$ and predictions $\hat{y} = \hat{y}_1, \dots, \hat{y}_n$ can be defined as

$$ \text{loss}(y, \hat{y}) = \frac{1}{mn} \sum_{j=1}^n\sum_{i=1}^m w_i {I}_{(y_j == i)}\text{log}(\hat{y}_j) $$

where ${I}_{(y_j == i)}$ is an indicator function which evaluates to 1 if $y_j == i$ and 0 otherwise.

One disadvantage is that it's not immediately obvious, given some value of the loss function, how good a particular value of the loss function is. However, it is easy to compare two values (lower is better).

Answer 2

In https://www.sciencedirect.com/science/article/pii/S004896971831163X?via%3Dihub, we have used the product of the sensitivity of class i (i.e., the ratio of data correctly classified for class i) obtained for each class. This summarizes these values into a single index, which ranges between 0 and 1 and to some extent is irrespective to data imbalance. This approach induces a good balance between the error comitted to each category because any value appreciably below 1 reduces significantly the performance. To relax this allowing a greater error in 1 o more categories you can simply use (in increasing order): the min, the geometric mean or the arithmetic mean. In https://ieeexplore.ieee.org/document/6940273/ and https://ieeexplore.ieee.org/abstract/document/5428802/ you can find these alternatives. Good luck. Rafael.

Answer 3

You have to use F1 score. A simple solution for that is to use confusion matrix. The way you can find F1 score for each class is simple. your true labels for each class can be considered as true predictions and the rest which are classified wrongly as the other classes should be added to specify the number of false predictions. For each class, you can find the F1 score. For more details take a look at F1-score per class for multi-class classification. You can take a look at this implementation.