3
$\begingroup$

I am looking for single-number evaluation method that can be used in multi-class classification tasks that take into account imbalanced data-sets. For instance, ROC-AUC defined by binary classifiers, is a single-number and takes into account imbalanced data-sets. On the other hand, accuracy is single-number, defined for multi-class classifiers and does not take into account imbalanced data-sets. Finally, the confusion matrix is defined for multi-class, takes that into account but is not single-number. Is there any evaluation method that satisfies the three conditions?

$\endgroup$
7
  • $\begingroup$ Are you familiar with F1 score? $\endgroup$ May 5, 2018 at 9:31
  • 1
    $\begingroup$ As far as I know it is for binary classifiers $\endgroup$ May 5, 2018 at 10:03
  • $\begingroup$ It can be extended like one vs. all $\endgroup$ May 5, 2018 at 10:21
  • $\begingroup$ But one vs all is not single-number. Is it? $\endgroup$ May 5, 2018 at 13:14
  • $\begingroup$ One vs. all is an extension of binary classification where you label the data to zeros and ones for each class for different classes. $\endgroup$ May 5, 2018 at 13:17

3 Answers 3

2
$\begingroup$

How about a weighted log-loss?

Lets say we have $m$ classes $c_1, \dots, c_m$. We can give each class $c_i$ a weight $w_i$ which is inversely proportional to the percentage of the dataset that belongs to $c_i$. Then, the loss for some data set with actual classes $y = y_1, \dots, y_n$ and predictions $\hat{y} = \hat{y}_1, \dots, \hat{y}_n$ can be defined as

$$ \text{loss}(y, \hat{y}) = \frac{1}{mn} \sum_{j=1}^n\sum_{i=1}^m w_i {I}_{(y_j == i)}\text{log}(\hat{y}_j) $$

where ${I}_{(y_j == i)}$ is an indicator function which evaluates to 1 if $y_j == i$ and 0 otherwise.

One disadvantage is that it's not immediately obvious, given some value of the loss function, how good a particular value of the loss function is. However, it is easy to compare two values (lower is better).

$\endgroup$
3
$\begingroup$

In https://www.sciencedirect.com/science/article/pii/S004896971831163X?via%3Dihub, we have used the product of the sensitivity of class i (i.e., the ratio of data correctly classified for class i) obtained for each class. This summarizes these values into a single index, which ranges between 0 and 1 and to some extent is irrespective to data imbalance. This approach induces a good balance between the error comitted to each category because any value appreciably below 1 reduces significantly the performance. To relax this allowing a greater error in 1 o more categories you can simply use (in increasing order): the min, the geometric mean or the arithmetic mean. In https://ieeexplore.ieee.org/document/6940273/ and https://ieeexplore.ieee.org/abstract/document/5428802/ you can find these alternatives. Good luck. Rafael.

$\endgroup$
2
$\begingroup$

You have to use F1 score. A simple solution for that is to use confusion matrix. The way you can find F1 score for each class is simple. your true labels for each class can be considered as true predictions and the rest which are classified wrongly as the other classes should be added to specify the number of false predictions. For each class, you can find the F1 score. For more details take a look at F1-score per class for multi-class classification. You can take a look at this implementation.

$\endgroup$
4
  • $\begingroup$ However, this is not single-number valued. Or is it? $\endgroup$ May 5, 2018 at 13:14
  • $\begingroup$ @DavidMasip it can be reported as the average! For each class it is reported as a separate number. $\endgroup$ May 5, 2018 at 13:15
  • $\begingroup$ I thought about that, but I wonder: do you have to weight your average with the number of examples or a plain average will do the Job? $\endgroup$ May 5, 2018 at 13:16
  • $\begingroup$ In your cost function you have to prioritize your rare class. You can also do that for your average F1 scores but I suggest reading them separately. It will help you do error analysis easily. $\endgroup$ May 5, 2018 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.