I have developed a content-based recommendation system and it is working fine. The input is a set of documents={d1,d2,d3,...,dn} and the output will be Top N similar documents for a given document output={d10,d11,d1,d8,...}. I eyeballed the results and found it to be satisfactory, the question I have is how do I measure the performance, accuracy of the system.

I did some research and found that recall, precision, and F1-score are used to evaluating the recommendation systems that predict user ratings. For this, we should no the original ratings and then the system should predict the ratings later we can plot the confusion matrix and then compute the aforementioned metric. However, in my case, I don't predict anything instead I measure the cosine similarity score sort it in descending order and pick the top N.

In this use case, how do I evaluate the system?



1 Answer 1


There's some confusion about different kinds of output and their corresponding evaluation:

  • One can consider the top N results as predicted positive, any result lower than N as predicted negative. In this option one can use binary classification evaluation measure: precision, recall, f1-score would be the standard measures in this case.
  • One can consider the ratings/scores assigned to the full set of results. In this case there are two options:
    • if the numerical results are comparable, e.g. same kind of rating, then standard regression evaluation measures can be used, for instance RMSE.
    • if not, then it's still possible to compare the order of the results. Spearman rank correlation is a common evaluation measure in this case.

It seems that in your case you could use either the classification or the ranking evaluation measures. Of course, any of these evaluation methods requires gold standard results in order to compare the predictions against them.

  • $\begingroup$ Yes, that is what even I found unfortunately I don't have the original labels. I have to label it manually or through some algorithm $\endgroup$
    – Raj
    Dec 11, 2020 at 6:11
  • $\begingroup$ @Raj labeling with an algorithm is not a good idea, since in this case you would have to evaluate the algorithm itself to know if its results are correct so you'd have the same problem. However you could try to find a similar dataset with annotations and evaluate your method on it. It's not as useful as labeling your own dataset but it's much easier and at least you can see how well your method works. $\endgroup$
    – Erwan
    Dec 11, 2020 at 11:20
  • $\begingroup$ This is a proprietary dataset one of its kind so I have to do it manually through a domain expert help to annotate the dataset then there will be questions how do you account for human relative error the solution is an algorithm which I am yet to find out. For now, I verified the results with help of the domain expert and certified it is working fine. $\endgroup$
    – Raj
    Dec 11, 2020 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.