Evaluating a IR system (Precision and Recall)

Question

I am studying by now IR system, in the field of valuation of IR system outputs related to a specific query but I need some help to understand it properly.

My book states that when an IR system has to be evaluated, we need a test document collection, a set of query examples, a valuation (relevant or not) for each couple of query/document, defined by experts in the field. So we need two measures to know quantitatively if a IR system is good: Precision and Recall.

My doubt is related to the following question: Do we use those two measures only if we are testing a IR system or not?

I'll explain: before we calculate Precision and Recall related to a specific query example (see above), we need to know how many elements belong to the relevant set, which is impossible if there isn't a valuation (relevant or not) for the query we are using. My book says we can increase Recall in a search engine by using the relevance feedback technique (query expansion and term reweighting): in this case do we assume the Recall value is unknown?

For example, everyday many documents are shared on the Internet and Google can find them. So it is impossibile to apply Recall and Precision to this scenario, in which information grows and there is no valuation for every new document for each specific query. It is also impossibile to predict all the possibile queries a user can do on a search engine.

Answer 1

My doubt is related to the following question: Do we use those two measures only if we are testing a IR system or not?

Technically the answer is no because precision and recall are used to evaluate not only IR systems but also many other tasks. However your question seems to be specific to IR so I'll assume that it's actually about the distinction between testing and evaluation:

• Testing a ML system consists in predicting the target variable for a set of instances given as input (in the case of supervised learning a "model" obtained from a previous stage of training is required as input as well). At this stage we don't know whether the predictions are correct or not.
• Evaluation is the process of assessing the quality of the predictions: it's done after obtaining the predictions from the testing stage, and it requires some form of "gold standard", i.e. data which says what is the correct answer for every instance.

In IR, the testing stage happens every time the system is run to find relevant documents based on a query.

• Naturally one wants at first to make sure that the system works properly and returns actually relevant documents, so the system needs to be evaluated, for instance with precision and recall using a dataset containing some queries and their relevant documents (gold standard).
• Once the quality is evaluated, the goal is to use the IR system (testing) without evaluating the results every time. Of course there's no evaluation so the performance measures (precision and recall) are not used.