3
$\begingroup$

I am looking for single-number evaluation method that can be used in multi-class classification tasks that take into account imbalanced data-sets. For instance, ROC-AUC defined by binary classifiers, is a single-number and takes into account imbalanced data-sets. On the other hand, accuracy is single-number, defined for multi-class classifiers and does not take into account imbalanced data-sets. Finally, the confusion matrix is defined for multi-class, takes that into account but is not single-number. Is there any evaluation method that satisfies the three conditions?

$\endgroup$
  • $\begingroup$ Are you familiar with F1 score? $\endgroup$ – Media May 5 '18 at 9:31
  • 1
    $\begingroup$ As far as I know it is for binary classifiers $\endgroup$ – David Masip May 5 '18 at 10:03
  • $\begingroup$ It can be extended like one vs. all $\endgroup$ – Media May 5 '18 at 10:21
  • $\begingroup$ But one vs all is not single-number. Is it? $\endgroup$ – David Masip May 5 '18 at 13:14
  • $\begingroup$ One vs. all is an extension of binary classification where you label the data to zeros and ones for each class for different classes. $\endgroup$ – Media May 5 '18 at 13:17
2
$\begingroup$

How about a weighted log-loss?

Lets say we have $m$ classes $c_1, \dots, c_m$. We can give each class $c_i$ a weight $w_i$ which is inversely proportional to the percentage of the dataset that belongs to $c_i$. Then, the loss for some data set with actual classes $y = y_1, \dots, y_n$ and predictions $\hat{y} = \hat{y}_1, \dots, \hat{y}_n$ can be defined as

$$ \text{loss}(y, \hat{y}) = \frac{1}{mn} \sum_{j=1}^n\sum_{i=1}^m w_i {I}_{(y_j == i)}\text{log}(\hat{y}_j) $$

where ${I}_{(y_j == i)}$ is an indicator function which evaluates to 1 if $y_j == i$ and 0 otherwise.

One disadvantage is that it's not immediately obvious, given some value of the loss function, how good a particular value of the loss function is. However, it is easy to compare two values (lower is better).

$\endgroup$
3
$\begingroup$

In https://www.sciencedirect.com/science/article/pii/S004896971831163X?via%3Dihub, we have used the product of the sensitivity of class i (i.e., the ratio of data correctly classified for class i) obtained for each class. This summarizes these values into a single index, which ranges between 0 and 1 and to some extent is irrespective to data imbalance. This approach induces a good balance between the error comitted to each category because any value appreciably below 1 reduces significantly the performance. To relax this allowing a greater error in 1 o more categories you can simply use (in increasing order): the min, the geometric mean or the arithmetic mean. In https://ieeexplore.ieee.org/document/6940273/ and https://ieeexplore.ieee.org/abstract/document/5428802/ you can find these alternatives. Good luck. Rafael.

$\endgroup$
2
$\begingroup$

You have to use F1 score. A simple solution for that is to use confusion matrix. The way you can find F1 score for each class is simple. your true labels for each class can be considered as true predictions and the rest which are classified wrongly as the other classes should be added to specify the number of false predictions. For each class, you can find the F1 score. For more details take a look at F1-score per class for multi-class classification. You can take a look at this implementation.

$\endgroup$
  • $\begingroup$ However, this is not single-number valued. Or is it? $\endgroup$ – David Masip May 5 '18 at 13:14
  • $\begingroup$ @DavidMasip it can be reported as the average! For each class it is reported as a separate number. $\endgroup$ – Media May 5 '18 at 13:15
  • $\begingroup$ I thought about that, but I wonder: do you have to weight your average with the number of examples or a plain average will do the Job? $\endgroup$ – David Masip May 5 '18 at 13:16
  • $\begingroup$ In your cost function you have to prioritize your rare class. You can also do that for your average F1 scores but I suggest reading them separately. It will help you do error analysis easily. $\endgroup$ – Media May 5 '18 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.