I am studying by now IR system, in the field of valuation of IR system outputs related to a specific query but I need some help to understand it properly.
My book states that when an IR system has to be evaluated, we need a test document collection, a set of query examples, a valuation (relevant or not) for each couple of query/document, defined by experts in the field. So we need two measures to know quantitatively if a IR system is good: Precision and Recall.
My doubt is related to the following question: Do we use those two measures only if we are testing a IR system or not?
I'll explain: before we calculate Precision and Recall related to a specific query example (see above), we need to know how many elements belong to the relevant set, which is impossible if there isn't a valuation (relevant or not) for the query we are using. My book says we can increase Recall in a search engine by using the relevance feedback technique (query expansion and term reweighting): in this case do we assume the Recall value is unknown?
For example, everyday many documents are shared on the Internet and Google can find them. So it is impossibile to apply Recall and Precision to this scenario, in which information grows and there is no valuation for every new document for each specific query. It is also impossibile to predict all the possibile queries a user can do on a search engine.