3
$\begingroup$

I'm searching for data sets for evaluating text retrieval quality.

TF-IDF is a popular similarity measure, but is it the best choice? And which variant is the best choice? Lucenes Scoring for example uses IDF^2, and IDF defined as 1+log(numdocs/(docFreq+1)). TF in lucene is defined as sqrt(frequency)...

Many more variants exist, including Okapi BM25, which is used by the Xapian search engine...

I'd like to study the different variants, and I'm looking for evaluation data sets. Thanks!

$\endgroup$
  • $\begingroup$ This is off topic in this Stack Exchange. Your post belongs in: opendata.stackexchange.com $\endgroup$ – sheldonkreger Nov 12 '14 at 23:32
  • $\begingroup$ This question appears to be off-topic because it is seeking a data set, and I believe that is more specifically on-topic at opendata.stackexchange.com $\endgroup$ – Sean Owen Nov 15 '14 at 14:28
1
$\begingroup$

TF IDF will give you the degree of measure of how relevant a document is to your query. owever, to evaluate your IR system you need to use metrics such as - Precision, Recall and F score.

Precision: Out of all the documents that your system retrieves, which ones are relevant? This measures how much noise there is in the output of your IR system.

Recall: Out of all the documents that are relevant, which ones did your system retrieve? This measures how much coverage does your IR system have?

It is possible to get 100% recall all the time by basically retrieving ALL documents from a collection for any query. However, the precision in this case will be very low.

It is possible to get a very high precision by hand modeling an IR system ti produce very accurate results. However, it would produce a very bad recall as there will not be coverage over all the documents.

So we need to measure F score- which is the harmonic mean between Precision and Recall Check out Chapter 8 of the Stanford IR book.

If you are looking for datasets only here are a few that are relevant:

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.