I have a text classification problem, where the "positive" examples are the minority. What metric is better to use for binary classification for this case - F1-score or AUC?
1 Answer
$\begingroup$
$\endgroup$
F1-score and AUC are two evaluation measures for binary classification but they are not comparable:
- F1-score measures the performance of a hard classifier, i.e. a system which predicts a class for every instance. Through precision and recall it compares for every instance the predicted class vs. the gold-standard class.
- AUC measures the performance of a soft classifier, i.e. a system which predicts a probability (or a score) for every instance. The difference is that the system doesn't decide which class the instance belongs to, so informally it can be seen as an "unfinished classifier". If one decides a threshold on the probability to separate the classes then it becomes a hard classifier.
Both are fine to be used with imbalanced data, that's not a reason to pick one or the other.
AUC is useful to study the general behaviour of a method without deciding a particular threshold. Sometimes the choice of a particular threshold can have a strong impact on performance, so using AUC avoids the issue completely. However strictly speaking AUC doesn't give the performance of a classifier.
