Performance Measures for Multi-Class Problems
For classification problems, classifier performance is typically defined according to the confusion matrix associated with the classifier. Based on the entries of the matrix, it is possible to compute sensitivity (recall), specificity, and precision. For a single cutoff, these quantities lead to balanced accuracy (sensitivity and specificity) or to the F1-score (recall and precision). To evaluate a scoring classifier at multiple cutoffs, these quantities can be used to determine the area under the ROC curve (AUC) or the area under the precision-recall curve (AUCPR).
All of these performance measures are easily obtainable for binary classification problems. Which measure is appropriate depends on the type of classifier.
F-Measure/F1-Score for Imbalanced Classification
Classification accuracy is widely used because it is one single measure used to summarize model performance. F-Score provides a way to combine both precision and recall into a single measure that captures both properties.
Alone, neither precision nor recall tells the whole story. We can have excellent precision with terrible recall, or alternately, terrible precision with excellent recall. F-measure provides a way to express both concerns with a single score.
Once precision and recall have been calculated for a binary or multiclass classification problem, the two scores can be combined into the calculation of the F-Measure. The traditional F measure is calculated as follows:
$\begin{equation}
\nonumber
F-Measure=\frac{2 * Precision * Recall}{Precision + Recall}
\end{equation}$
For a multi-class classification problem, we don’t calculate an overall F-1 score. Instead, we calculate the F-1 score per class in a one-vs-rest manner.
Here are simple Python functions that do exactly that:
def precision(y_true, y_pred):
i = set(y_true).intersection(y_pred)
len1 = len(y_pred)
if len1 == 0:
return 0
else:
return len(i) / len1
def recall(y_true, y_pred):
i = set(y_true).intersection(y_pred)
return len(i) / len(y_true)
def f1(y_true, y_pred):
p = precision(y_true, y_pred)
r = recall(y_true, y_pred)
if p + r == 0:
return 0
else:
return 2 * (p * r) / (p + r)
For imbalanced classes, I would suggest to go with Weighted F1-Score or Average AUC/Weighted AUC or Micro and macro averages of the F1-score
Weighted F1-Score was calculated by F1 score for each class independently but when it adds them together uses a weight that depends on the number of true labels of each class:
$F1_{class1}*W_1+F1_{class2}*W_2+\cdot\cdot\cdot+F1_{classN}*W_N$
therefore favoring the majority class.
Micro and macro averages of the F1-score
Micro and macro averages represent two ways of interpreting confusion matrices in multi-class settings. Here, we need to compute a confusion matrix for every class $g_i \in G = \{1, \ldots, K\}$ such that the $i-th$ confusion matrix considers class $g_i$ as the positive class and all other classes $g_j$ with $j≠i$ as the negative class. Since each confusion matrix pools all observations labeled with a class other than $g_i$ as the negative class, this approach leads to an increase in the number of true negatives, especially if there are many classes.
To exemplify why the increase in true negatives is problematic, imagine there are 10 classes with 10 observations each. Then the confusion matrix for one of the classes may have the following structure:
Prediction/Reference Class 1 Other Class
Class 1 8 10
Other Class 2 80
Based on this matrix, the specificity would be $\frac{80}{80 + 10} = 88.9\%$ although class 1 was only correctly predicted in 8 out of 18 instances (precision 44.4%). Thus, since the negative class is predominant, the specificity becomes inflated. Thus, micro- and macro averages are only defined for the F1-score and not for the balanced accuracy, which relies on the true negative rate.
In the following, we will use $TP_i$, $FP_i$, and $FN_i$ to respectively indicate true positives, false positives, and false negatives in the confusion matrix associated with the $i-th$ class. Moreover, let precision be indicated by $P$ and recall by $R$.
The micro average has its name from the fact that it pools the performance over the smallest possible unit (i.e. over all samples):
$\begin{align*}
P_{\rm{micro}} &= \frac{\sum_{i=1}^{|G|} TP_i}{\sum_{i=1}^{|G|} TP_i+FP_i} \\
R_{\rm{micro}} &= \frac{\sum_{i=1}^{|G|} TP_i}{\sum_{i=1}^{|G|} TP_i + FN_i}
\end{align*}$
The micro-averaged precision, $P_{micro}$, and recall, $R_{micro}$, give rise to the micro F1-score:
$F1_{\rm{micro}} = 2 \frac{P_{\rm{micro}} \cdot R_{\rm{micro}}}{P_{\rm{micro}} + R_{\rm{micro}}}$
If a classifier obtains a large $F1_{micro}$, this indicates that it performs well overall. The micro-average is not sensitive to the predictive performance for individual classes. As a consequence, the micro-average can be particularly misleading when the class distribution is imbalanced.
The macro average has its name from the fact that it averages over larger groups, namely over the performance for individual classes rather than observations:
$\begin{align*}
P_{\rm{macro}} &= \frac{1}{|G|} \sum_{i=1}^{|G|} \frac{TP_i}{TP_i+FP_i} = \frac{\sum_{i=1}^{|G|} P_i}{|G|}\\
R_{\rm{macro}} &= \frac{1}{|G|} \sum_{i=1}^{|G|} \frac{TP_i}{TP_i + FN_i} = \frac{\sum_{i=1}^{|G|} R_i}{|G|}
\end{align*}$
The macro-averaged precision and recall give rise to the macro F1-score:
$F1_{\rm{macro}} = 2 \frac{P_{\rm{macro}} \cdot R_{\rm{macro}}}{P_{\rm{macro}} + R_{\rm{macro}}}$
If $F1_{macro}$ has a large value, this indicates that a classifier performs well for each individual class. The macro-average is, therefore, more suitable for data with an imbalanced class distribution.
References:
