0
$\begingroup$

I understand precision at k and recall at k. It is a more useful metric for evaluating the success of a binary classifier when the positive class is overwhelmingly out-weighed by the negative class.

I'm wondering how to choose an appropriate "k" value. According to resources like this the recall at "k" is bounded by the number of positive examples, so it is not a useful metric to use when evaluating the success of severely imbalanced classes.

It also seem to me like this: Precision at K is also limited by the number of positive examples at K. Saw we have 100 examples total, and only 3 are positive examples. Say we rank these

Scenario 1: we choose k=10. Then, the precision at K can at most be 3/10 = 0.3. And, the recall at k will be 0.03 because there are 3 in the entire dataset of 100.

Scenario 2: we choose k=3. Then, the precision at K would be 3/3 = 1.0 !!!And, the recall at k will STILL be 3/100 = 0.03.

Even though our binary classifier is performing perfectly, it's perfect performance only is reflected when we choose k=3?

So, my question how do I choose K correctly?

$\endgroup$

2 Answers 2

1
$\begingroup$

Precision and Recall can be dubious at times. It depends on the point on the ROC Curve. Therefore, pick the point as per your need. The best way is to give more information about your results; i.e., Accuracy measure and Confusion matrix.

$\endgroup$
3
  • 1
    $\begingroup$ "Precision and Recall can be dubious at times." Can you explain what you mean by this? $\endgroup$
    – kbrose
    Jun 15, 2017 at 18:03
  • $\begingroup$ It depends on the point of interest on the ROC curve and they have a trade-off between them. $\endgroup$ Jun 15, 2017 at 20:49
  • $\begingroup$ Couldn't you just as easily say the Specificity and Recall depend on the point of interest on the PR curve? In some fields, Precision is much more important information than specificity. $\endgroup$
    – kbrose
    Jun 16, 2017 at 15:19
1
$\begingroup$

If all of your top num_positive terms are positive class then your precision@k should be 1.0 for all k.

The denominator should be min(num_positive, num_looked_at), not num_looked_at.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.