3
$\begingroup$

I am studying by now IR system, in the field of valuation of IR system outputs related to a specific query but I need some help to understand it properly.

My book states that when an IR system has to be evaluated, we need a test document collection, a set of query examples, a valuation (relevant or not) for each couple of query/document, defined by experts in the field. So we need two measures to know quantitatively if a IR system is good: Precision and Recall.

My doubt is related to the following question: Do we use those two measures only if we are testing a IR system or not?

I'll explain: before we calculate Precision and Recall related to a specific query example (see above), we need to know how many elements belong to the relevant set, which is impossible if there isn't a valuation (relevant or not) for the query we are using. My book says we can increase Recall in a search engine by using the relevance feedback technique (query expansion and term reweighting): in this case do we assume the Recall value is unknown?

For example, everyday many documents are shared on the Internet and Google can find them. So it is impossibile to apply Recall and Precision to this scenario, in which information grows and there is no valuation for every new document for each specific query. It is also impossibile to predict all the possibile queries a user can do on a search engine.

$\endgroup$
2
$\begingroup$

My doubt is related to the following question: Do we use those two measures only if we are testing a IR system or not?

Technically the answer is no because precision and recall are used to evaluate not only IR systems but also many other tasks. However your question seems to be specific to IR so I'll assume that it's actually about the distinction between testing and evaluation:

  • Testing a ML system consists in predicting the target variable for a set of instances given as input (in the case of supervised learning a "model" obtained from a previous stage of training is required as input as well). At this stage we don't know whether the predictions are correct or not.
  • Evaluation is the process of assessing the quality of the predictions: it's done after obtaining the predictions from the testing stage, and it requires some form of "gold standard", i.e. data which says what is the correct answer for every instance.

In IR, the testing stage happens every time the system is run to find relevant documents based on a query.

  • Naturally one wants at first to make sure that the system works properly and returns actually relevant documents, so the system needs to be evaluated, for instance with precision and recall using a dataset containing some queries and their relevant documents (gold standard).
  • Once the quality is evaluated, the goal is to use the IR system (testing) without evaluating the results every time. Of course there's no evaluation so the performance measures (precision and recall) are not used.
| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you. So without any gold standard, testing is impossibile to do, I guess. $\endgroup$ – Angelo Giannuzzi Dec 31 '19 at 19:06
  • $\begingroup$ My book says: "In order to improve recall values, we need to reformulate queries by using query expansion, term reweighting or Rocchio's formula": I guess this statement refers qualitatively to the concept of recall value. If we have not idea of which are the relevant documents for the specific query we are using (we can do this only if we are using a gold standard), it is impossibile to talk about recall value, isn't it? $\endgroup$ – Angelo Giannuzzi Dec 31 '19 at 19:14
  • $\begingroup$ @AngeloGiannuzzi without any gold standard only evaluating is impossible. Yes, as you say they are talking about a method which is known to increase recall, it doesn't mean that the method requires to be evaluated every time. It should be understood in the same way as saying that a healthy diet increases life expectancy: you don't need to measure the diet and age of every single person when they die, it's a general statement. $\endgroup$ – Erwan Jan 1 at 0:01
  • $\begingroup$ Thank you, Erwan. You have solved my doubt. $\endgroup$ – Angelo Giannuzzi Jan 1 at 2:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.