2
$\begingroup$

On Wikipedia there is a practical example of calculating Precision and Recall:

When a search engine returns 30 pages, only 20 of which are relevant, while failing to return 40 additional relevant pages, its precision is 20/30 = 2/3, which tells us how valid the results are, while its recall is 20/60 = 1/3, which tells us how complete the results are.

I absolutely don't understand how one can use the Precision and Recall in real/life scenario of total number of relevant documents is needed.

For example, In my scenario, I have a set of about 9000 collected documents and I am creating a recommender system with several algorithms (like Tf-idf, Doc2Vec, LDA...). It has to recommend the TOP 20 most similar recommendations (articles) based on one selected article. Since I am not going to count the number of all relevant articles manually in 9000 documents for every recommender query, what is a relevant way to estimate the total number of relevant articles so that I can calculate Recall and then proceed to calculate Average Precision?

The only information I found about this problem are this lecture notes where they suggest to create pool of the result:

There are several ways of creating a pool of relevant records: one method is to use all the relevant records found from different searches, another is to manually scan several journals to identify a set of relevant papers.

But I'm trying to find more information on this method of "pools" elsewhere.

Common sense is telling me that this can be a valid approach: To take, say, 50 random documents and manually count the number of relevant documents in that random sample and estimate the total number of relevant documents from that. Can this be a valid approach? I imagine I could do this for a few recommendation results (although it would be a bit time consuming) or have some test users selected.

Improve this question
$\endgroup$
3
  • $\begingroup$ Welcome to DataScienceSE. Note that precision and recall are evaluation measures for supervised learning. In the standard setting of classification, the labelled data (training set and test set for evaluation) is fully annotated, with the appropriate class known for every instance. Obviously it's easy to calculate recall in this case. I'm no expert for evaluation in the case of recommendation/ranking, but clearly if you can't have the full data annotated you just can't calculate recall. $\endgroup$
    – Erwan
    Apr 27, 2022 at 14:07
  • $\begingroup$ OP, the lang seems unclear. You have articles within the document? Does that mean that those articles are the [denominator] documents that the Wiki example refers to? $\endgroup$
    – sandyp
    Apr 28, 2022 at 16:42
  • $\begingroup$ @sandyp Texts of news articles are my documents. $\endgroup$
    – Banik
    Apr 29, 2022 at 8:22

1 Answer 1

Reset to default
1
$\begingroup$

I think the answer to my question are "at k" ("@k") variants of above mentdioned methods: precision@k, recall@k, precision@k etc. I need to set the threshold to let's say TOP 20 (k=20) examples and then evaluate the results of precision and recall (by hand myself or by test users decision who will decide whether the recommendation is relvant or irrelevant). I found good practical examples here for anyone interested in the same problem at queirozf.com.

For example:

Recall @8 = true_positives@8 / (true_positives@8) + (false_negatives@8))
Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.