When I am training and evaluating classifiers or hyperparameter tuning I don't like to look at precision and recall metrics alone because those numbers depend on a threshold that I will set afterwards to cut the output probabilities and make the final prediction. This threshold usually defaults to .5. But that depends entirely on the problem and I always end up changing it. I like to decide its value at the end, once I have selected the ~3 best models from the previous evaluation phase. Sometimes I need a conservative classifier and sometimes a more relaxed one. So before going into that, I need something to use to select the best models and the metric I look at the most is the area under the precision-recall curve. If I had to choose only one metric to look at it would be that. However it is not that popular and I see people using precision and recall numbers alone (threshold=0.5) to evaluate their models.

Why isn't it used to grid-search/model selection more often? Maybe I am missing an important drawback?

  • 1
    $\begingroup$ When a decision threshold is involved the area under P/R curve is indeed more informative then just P or R or F1. But a more standard metric is the area under the ROC curve, which is widely used to report the final result (on test set). Maybe it is less used for grid search since it's more complicated to calculate. $\endgroup$ – user12075 Sep 22 '18 at 16:14
  • $\begingroup$ Thanks @user12075. Could you elaborate on why is it more complicated, please? $\endgroup$ – hipoglucido Oct 7 '18 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.