What is the best metric to evaluate highly imbalanaced binary classifiction? (such as fraud detection in credit card)

Question

What is the best metric to evaluate highly imbalanaced binary classifiction? (such as fraud detection in credit card?

I have examining several metrics precision recall F1 lassification Report (macro avg,weighted avg), ROC, AUC,.. but I do not know what is more acceptable to evaulate highly imbalanced binary classification such as credit card fraud detection https://www.kaggle.com/mlg-ulb/creditcardfraud

Answer 1

It would depend on use case that you are addressing. Say in a healthcare where you are using it for medical diagnosis recall might take precedence since false negatives can be quite detrimental there however in the use case you mentioend fraud detection both false positives and false negatives can have dire impact you would look at F1-score

Answer 2

Simple precision/recall/F-score is perfectly suitable for imbalanced data. It should be computed on the minority class of course.

• precision says how often the system is correct when predicting an instance in the minority class.
• recall says how often the system detects an instance which belongs to the minority class (this is usually the hardest part with imbalanced data)
• As usual F-score can be used as a "summary metric" of both precision and recall.

ROC and AUC (which are also based on precision/recall) can be used only with a soft classifier, i.e. when the system predicts a numerical value instead of a binary label.

Macro/micro/weighted average (usually over F-score) are irrelevant: averaging over the two classes would take into account the majority class which is very easy to predict, so it would mask the useful information about the minority class.

Answer 3

F1 score is my main metric for model comparison, especially when evaluating models using imbalanced datasets. Accuracy is clearly out of the question, and ROC-AUC is also skewed under class imbalance (1). If you favor higher precision or higher recall you can also adjust the beta parameter in the generalized F-beta score (2), but I'm usually looking for a balance, so beta is always 1 in my cases. The Matthews Correlation Coefficient (MCC) is also highly recommended by some, though I haven't used it personally.