2
$\begingroup$

I have a text classification problem, where the "positive" examples are the minority. What metric is better to use for binary classification for this case - F1-score or AUC?

$\endgroup$
2
$\begingroup$

F1-score and AUC are two evaluation measures for binary classification but they are not comparable:

  • F1-score measures the performance of a hard classifier, i.e. a system which predicts a class for every instance. Through precision and recall it compares for every instance the predicted class vs. the gold-standard class.
  • AUC measures the performance of a soft classifier, i.e. a system which predicts a probability (or a score) for every instance. The difference is that the system doesn't decide which class the instance belongs to, so informally it can be seen as an "unfinished classifier". If one decides a threshold on the probability to separate the classes then it becomes a hard classifier.

Both are fine to be used with imbalanced data, that's not a reason to pick one or the other.

AUC is useful to study the general behaviour of a method without deciding a particular threshold. Sometimes the choice of a particular threshold can have a strong impact on performance, so using AUC avoids the issue completely. However strictly speaking AUC doesn't give the performance of a classifier.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.