I've seen information retrieval systems that return some results from a query, and then the user rates these results as either "relevant" or "not relevant".

What can you do if you do not have user feedback?

E.g. suppose your system returns some ranked results from a query. Suppose you have no pre-defined notion of what is relevant, and suppose you cannot receive any kind of user feedback. What can you do?

This is important, because information retrieval evaluation metrics are based on relevance. Maybe it isn't possible to define relevance without user feedback, if so, maybe some information retrieval evaluation metrics without dependency on user feedback can be suggested?

  • 1
    $\begingroup$ This is quite relevant to my own recent question: datascience.stackexchange.com/questions/64358/… Maybe an answer there will be helpful to you if someone responds. Or maybe if someone answers your question, it will be helpful to me! $\endgroup$
    – Data
    Commented Dec 7, 2019 at 16:56

1 Answer 1


There is no formal definition for the concept of relevance, because relevance depends completely on the context and is therefore highly subjective. This is why the best way (some might say the only way) to evaluate relevance is to actually ask users what is relevant for them.

For any ML-based task, one needs to design a proper evaluation framework in order to control and measure the quality of the results. Naturally the evaluation method should be chosen so that it reflects as much as possible the level of quality with respect to the goal of the task, i.e. what one would intuitively expect from it. Evaluation metrics are almost always simplified indicators of this "level of quality", so what matters is how well they correlate with what a user would expect from the system: sometimes even a perfectly standard evaluation measure might not to be suited to the goal of the task.

My point is that evaluation is a matter of analysis and design. There are infinitely many options, but the point is to select the most appropriate one for the job. Here are some of these options:

  • The ideal case is to have annotated data (e.g. user feedback) directly suited to the data and the task: then it's just a matter of counting how often the prediction is correct.
  • It's common to evaluate a system against another annotated dataset X, assuming that the task is similar enough so if the system works well on X then it will work well on the real dataset.
  • Another less than ideal way is to evaluate against the predictions of another reference system X: in this case X is considered the gold standard, so there is no way for the tested system to perform better than X.
  • Indirect evaluation: if there is another task being performed at a later stage with the predictions, and this task can be evaluated more easily than the IR task itself.
  • Heuristics: that would be the least reliable kind of evaluation, but it's better than nothing. It ranges from simply counting the number of words in common between the query and top N results to developing complex methods using third-party resources.
  • $\begingroup$ Thanks for such a thorough response. I've noticed you provided an answer to the question linked in the comments of this question. Would you say that their ad-hoc method of evaluating relevance is an example of a "heuristic"? $\endgroup$
    – DataGuy
    Commented Dec 7, 2019 at 18:36
  • $\begingroup$ @DataGuy Yes I think so. $\endgroup$
    – Erwan
    Commented Dec 7, 2019 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.