For the imbalanced datasets:
- Can we say the Precision-Recall curve is more informative, thus accurate, than ROC curve?
- Can we rely on F1-score to evaluate the skillfulness of the resulted model in this case?
$\begingroup$
$\endgroup$
6
-
$\begingroup$ Yes. Precision-Recall curve is absolutely reliable for imbalanced datasets when compared with ROC curve. F-1 score is so far one of the most reliable metric to measure the performance of a model. You are thinking in the right direction. $\endgroup$– Adhira DeogadeMay 7, 2020 at 18:07
-
1$\begingroup$ @AdhiraDeogade Thank you very much for your note. very valuable $\endgroup$– DaveMay 7, 2020 at 18:17
-
$\begingroup$ Sure. Keep asking. $\endgroup$– Adhira DeogadeMay 7, 2020 at 18:18
-
$\begingroup$ @AdhiraDeogade Reliable with respect to what? It seems there is another measure apart from Precision-Recall/ROC/F1 that you are using to judge these. $\endgroup$– JTHMay 7, 2020 at 23:35
-
2$\begingroup$ I just think it is reckless to say "measure A is better than measure B" when no evidence or benchmark is given. I think the value of one measure over another additionally depends on what the practitioner intends to do with the model, unless we are just talking about the intrinsic value of one model over another, then I believe proper scoring rules should be used. $\endgroup$– JTHMay 8, 2020 at 2:29
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
Precision-recall curves are argued to be more useful than ROC curves in "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets" by Saito and Rehmsmeier. They argue that ROC might lead to the wrong visual interpretation of specificity.
F1-score equally balances precision and recall. In some domains it might be more useful to more heavily weight precision (F < 1) or recall (F > 1).
answered Nov 29, 2020 at 4:14