0
$\begingroup$

I understand precision at k and recall at k. It is a more useful metric for evaluating the success of a binary classifier when the positive class is overwhelmingly out-weighed by the negative class.

I'm wondering how to choose an appropriate "k" value. According to resources like this the recall at "k" is bounded by the number of positive examples, so it is not a useful metric to use when evaluating the success of severely imbalanced classes.

It also seem to me like this: Precision at K is also limited by the number of positive examples at K. Saw we have 100 examples total, and only 3 are positive examples. Say we rank these

Scenario 1: we choose k=10. Then, the precision at K can at most be 3/10 = 0.3. And, the recall at k will be 0.03 because there are 3 in the entire dataset of 100.

Scenario 2: we choose k=3. Then, the precision at K would be 3/3 = 1.0 !!!And, the recall at k will STILL be 3/100 = 0.03.

Even though our binary classifier is performing perfectly, it's perfect performance only is reflected when we choose k=3?

So, my question how do I choose K correctly?

$\endgroup$
1
$\begingroup$

Precision and Recall can be dubious at times. It depends on the point on the ROC Curve. Therefore, pick the point as per your need. The best way is to give more information about your results; i.e., Accuracy measure and Confusion matrix.

| improve this answer | |
$\endgroup$
  • $\begingroup$ "Precision and Recall can be dubious at times." Can you explain what you mean by this? $\endgroup$ – kbrose Jun 15 '17 at 18:03
  • $\begingroup$ It depends on the point of interest on the ROC curve and they have a trade-off between them. $\endgroup$ – Vivek Khetan Jun 15 '17 at 20:49
  • $\begingroup$ Couldn't you just as easily say the Specificity and Recall depend on the point of interest on the PR curve? In some fields, Precision is much more important information than specificity. $\endgroup$ – kbrose Jun 16 '17 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.