I understand precision at k and recall at k. It is a more useful metric for evaluating the success of a binary classifier when the positive class is overwhelmingly out-weighed by the negative class.
I'm wondering how to choose an appropriate "k" value. According to resources like this the recall at "k" is bounded by the number of positive examples, so it is not a useful metric to use when evaluating the success of severely imbalanced classes.
It also seem to me like this: Precision at K is also limited by the number of positive examples at K. Saw we have 100 examples total, and only 3 are positive examples. Say we rank these
Scenario 1: we choose k=10. Then, the precision at K can at most be 3/10 = 0.3. And, the recall at k will be 0.03 because there are 3 in the entire dataset of 100.
Scenario 2: we choose k=3. Then, the precision at K would be 3/3 = 1.0 !!!And, the recall at k will STILL be 3/100 = 0.03.
Even though our binary classifier is performing perfectly, it's perfect performance only is reflected when we choose k=3?
So, my question how do I choose K correctly?