6
$\begingroup$

Normally, I am familiar with precision and recall evaluation metrics but as you know recall@k and precision@k are different things and used in ranking evaluations especially recommendation systems.

I checked many sources, I understood everything I could not understand a point.

One more thing,

Every source is different between each other in terms of calculation ( 1 , 2, 3, 4 )

let's get this example

enter image description here

I'll give you an example so you can explain to me

Let's say we have 5 users. We are trying to give location recommendations for the next visit to each user. We are analyzing users' historical check-in data and we are giving recommendation for the next visit.

User 1 is visiting: Museum1, Park1, Night Club, "?" (What is next)

We are trying to find the next visiting locations. Let's say our ground truth "Restaurant"

How can I calculate precision@5 and recall@5?

Extra: This youtube video is explaining very good (go to 51:45 on video)

What is 5-6 relevant item means? If we are giving recommendation it should be just 1 item that is gonna be relevant for the user. They are trying to make a movie recommendation but they are saying we have 5 relevant movies. What is that mean?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

A quick answer.

Note: Checking the references I could access fully, there are no discrepancies between the definitions as long as the terms are translated correctly.

Some definitions:

Relevant items: Are items the user(s) have themselves established as relevant with their actions.

Recommended items: Are items which the recomendation system predicts will be relevant to the user(s).

Concerning your example:

Let’s understand the definitions of recall@k and precision@k, assume we are providing $5$ recommendations in this order — 1 0 1 0 1, where 1 represents relevant and 0 irrelevant. So the precision@k at different values of k will be precision@3 is $2/3$, precision@4 is $2/4$, and precision@5 is $3/5$. The recall@k would be, recall@3 is $2/3$, recall@4 is $2/3$, and recall@5 is $3/3$.

Reference: Precision and recall at k for recommender systems

Improve this answer
$\endgroup$
6
  • $\begingroup$ My question is. For example, we are trying to recommend locations for users. And we are providing 5 recommendations like ( location 1, location 2, location 3, location 4, location 5) Which one is relevant here. Only 1 location is correct in ground truth. ? $\endgroup$
    – drorhun
    Mar 28, 2021 at 22:44
  • $\begingroup$ If I say location1 and location 3 relevant for this user, I can do the calculations. How is it to have more than 1 relevant, I don't understand it $\endgroup$
    – drorhun
    Mar 28, 2021 at 22:46
  • $\begingroup$ There is a confusion over relevant vs recommended. Relevance is determined by the user, whereas recommended by the system. The system can very well recommend more than one item, in fact this is the norm (eg Amazon, Youtube, ..). Furthermore recommendations should introduce some novelty to the user, that is recommend items that the user has not yet identified as relevant and may not identify as such even in the future, because re-iterating always items the user has already found relevant becomes a triviality. $\endgroup$
    – Nikos M.
    Mar 29, 2021 at 7:49
  • $\begingroup$ I got very close to the answer. If you answer this question, I think I will confirm your answer. Users determine the relevant items. I have 100 users and each user visited 100 places. I split this dataset as training and test and my system learned the behavior of users. For the next move, I am trying to give location recommendations to each user. I said go to the next location central park for the first user. The user's current preference is not central park in the ground truth. My accuracy is 0 here. I recommend 5 locations for Precision @ 5. What is 5 relevant locations to calculate precisio $\endgroup$
    – drorhun
    Mar 29, 2021 at 8:18
  • $\begingroup$ Relevance is determined by the user. So for example using some rules of inference your system has ranked 5 possible locations this user should visit next. Some of these the user has already visited (these are the relevant), the rest are not (at this time) relevant, only recommended. $\endgroup$
    – Nikos M.
    Mar 29, 2021 at 8:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.