1
$\begingroup$

When evaluating the performance of a multiclass classification problem, on a highly imbalanced dataset, what is the most robust metric for this purpose?

I read a paper that states:

"Average precision is a robust metric in the presence of class imbalance since it excludes the ‘true negatives’ constituent in specificity, focusing instead on precision, or positive predictive value."

The confusing part, for me, is the applied methodology in this paper. They used balancing techniques (SMOTE, class_weigh, random sampling) to oversample minority classes, but they are still concerned about the evaluation metrics and 'true negatives'.

$\endgroup$

2 Answers 2

3
$\begingroup$

For evaluating the classification of a highly imbalanced, there are several measures that you may consider. Remember, that in such a problem we would prefer a measure that is not biased towards one of the classes but gives similar importance to both classes.

Methods for evaluating classifiers could be divided into two types: singular assessment measures where the evaluation is given for a specific threshold (i.e. trade-off) and area-under-curve (AUC) analyses where a range of thresholds is considered.

The F-measure combines precision and recall. Precision is sensitive to data distribution as it combines values from the two rows of the confusion-matrix hence, as the difference between the numbers of negative and positive instances gets larger, the influence of the positive class (minority) gets smaller. Recall, on the other hand, is not sensitive to data distribution since it does not refer to the negative class at all. Affected by the precision, the F-measure must be sensitive to class distribution as well. The F-measure gives higher importance to the correct classification of positive instances, a change in the TNR value will influence the F-measure value less than a change in TPR. This is explained by the impact of the recall, which considers only the correct classification of positive class instances.

The Geometric Mean (G-measure, GM) combines TPR and TNR, denoting the accuracy rates measured on both classes separately. The GM is independent of the class sizes, indicating the relative part of correctly classified instances in each class regardless of the degree of imbalance. That is, the GM is not influenced by the imbalance between the classes, The GM relates to the performance on both classes with the same importance since it is a harmonic mean of the TPR & TNR and their effect is symmetrical. Higher values of GM are achieved when the accuracies on both classes are high and similar. For example, a higher GM value is accomplished when accuracies on both classes are equal to 0.5 than when one is 0.6 and the other 0.4.

The ROC-AUC is based on the TPR and FPR, which is 1-TNR, and hence have the same reasoning for suiting such problems as the GM.

The PR-AUC, precision-recall curve, are also useful for highly imbalanced data and also is robust for changes in the balance of classes (addition of samples). I've found this article very useful for understanding AUC measures and the benefits of each one of them.

The AUC analysis provides a more generic evaluation however, in most cases it will not give the best solution when a specific trade-off is to be considered. The AUC is only relevant if your classifier output is not discrete but continues.

For more reading and comparisons you may use Prof. Haibo He lectures which are very comprehensive and useful.

$\endgroup$
1
  • $\begingroup$ GM is not influenced by class imbalance, and also you stated that ROC-AUC also have the same reasoning for suiting such problems as the GM, but isn't ROC-AUC sensitive to class imbalance? $\endgroup$
    – tkarahan
    May 31, 2021 at 9:07
0
$\begingroup$

F1-score or G-measure should be fine for imbalanced dataset.

To evaluate model to negative prediction consider negative predictive value or specificity.

Source: https://en.wikipedia.org/wiki/Sensitivity_and_specificity

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.