Consider the problem of learning to rank for Google-like searching - i.e., learning to return a good ordering of URL's when given a query. Most (if not all) current evaluation metrics for this problem are URL-based. Current evaluation metrics like Mean Reciprocal Rank (MRR) and Discounted Cumulative Gain (DCG) sum up weighted relevance scores for the pages in a ranked list. If a ranked list contains n pages, for instance, these metrics give a sum of n terms.

From my own literature review, it seems no metrics incorporate words. Although some theoretically incorporate "information nuggets", none incorporate "information nuggets=words" in practice. I have not found recent or past literature that concretely utilizes words, phrases, or anything similar in scoring ranked lists. Since I haven't found past literature, it does not seem that word-based metrics were attempted once and then replaced with coarser URL-based metrics. Documents contain words, as do the snippets returned in a Google-like ranked list, so it seems natural that word-based metrics should have been attempted. Thoughts?

(Note: One obvious objection is that TREC-based assessment of "word relevance" is a vague and even expensive endeavor (asking assessors to mark for words for relevance). But there's other ways to get relevance feedback that ignore this problem, such as using the dwell time on pages as a proxy.)

  • $\begingroup$ No metrics include words? en.wikipedia.org/wiki/Cosine_similarity has a similarity that can be used to rank. I don't follow the question. $\endgroup$ – paparazzo Nov 16 '15 at 18:04
  • $\begingroup$ Good question! Though cosine is for generating the features for ranking algorithms. Not for the evaluation of performance. $\endgroup$ – Matt Nov 16 '15 at 19:24
  • $\begingroup$ I don't follow but not important. I can tell you that I use tf-idf cosine similarity to rank search results. $\endgroup$ – paparazzo Nov 16 '15 at 19:53

Using a word-based metric would explicitly favor word-level retrieval methods. The theory is that (just as you suggest with dwell time), the URL-level metric measures more directly the desired result.

More concretely, consider a search of "alcohol from potatoes." Assume we have two pages:

1) A page that is simply a grocery list (containing "alcohol" and "potatoes") 
2) A page that describes vodka or moonshine which only uses the words "ethanol"
   and "potatoes" but not "alcohol."

A word-level metric would prefer the first result, since it contains both alcohol and potatoes.. A URL-level metric would likely label the second result as more relevant, which would allow a model to form associations beyond simple word-matching (i.e. it could learn that "alcohol" and "vodka" are associated).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.