0
$\begingroup$

If a 3-class classifier returns a length-3 vector of probabilities, e.g. [0.1, 0.85, 0.05] for classes A, B, and C respectively (strongly indicating B), does it make sense to use F1@3 (or precision@3 or recall@3) as a metric?

From my understanding of precision and recall "at k" it seems appropriate for models that return arbitrary numbers of results.

But for models that return a fixed quantity (like softmax or rankings), the number of output values is always equal to the number of classes. Does it ever make sense to use F1@k for this kind of model?

$\endgroup$

1 Answer 1

2
$\begingroup$

If your use case is producing probabilities of 3 classes, you should use multiclass precision/recall/f1, rather than the @k versions.

value@k type metrics are typically used in retrieval/ranking problems where the model output is used to rank order some set of things.

For example, a simple recommendation system might be a binary classifier. 1 means recommend to user, 0 means don't recommend to user. The model outputs a probability value between 0 and 1, which is used to rank order possible items to show.

Since the user will only see the top k subset of items (ie top 10 items sent to user feed), we want to compute our metrics on just the top subset as this is most relevant to the use case.

$\endgroup$
1
  • $\begingroup$ Thanks for pointing out the @k is for retrieval/ranking problems. That point tends to be implicit or not even mentioned in literature about precision and recall. $\endgroup$ Oct 11, 2023 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.