Why are there currently no content-based evaluation metrics for information retrieval?

Question

Consider the problem of learning to rank for Google-like searching - i.e., learning to return a good ordering of URL's when given a query. Most (if not all) current evaluation metrics for this problem are URL-based. Current evaluation metrics like Mean Reciprocal Rank (MRR) and Discounted Cumulative Gain (DCG) sum up weighted relevance scores for the pages in a ranked list. If a ranked list contains n pages, for instance, these metrics give a sum of n terms.

From my own literature review, it seems no metrics incorporate words. Although some theoretically incorporate "information nuggets", none incorporate "information nuggets=words" in practice. I have not found recent or past literature that concretely utilizes words, phrases, or anything similar in scoring ranked lists. Since I haven't found past literature, it does not seem that word-based metrics were attempted once and then replaced with coarser URL-based metrics. Documents contain words, as do the snippets returned in a Google-like ranked list, so it seems natural that word-based metrics should have been attempted. Thoughts?

(Note: One obvious objection is that TREC-based assessment of "word relevance" is a vague and even expensive endeavor (asking assessors to mark for words for relevance). But there's other ways to get relevance feedback that ignore this problem, such as using the dwell time on pages as a proxy.)

Answer 1

Using a word-based metric would explicitly favor word-level retrieval methods. The theory is that (just as you suggest with dwell time), the URL-level metric measures more directly the desired result.

More concretely, consider a search of "alcohol from potatoes." Assume we have two pages:

1) A page that is simply a grocery list (containing "alcohol" and "potatoes") 
2) A page that describes vodka or moonshine which only uses the words "ethanol"
   and "potatoes" but not "alcohol."

A word-level metric would prefer the first result, since it contains both alcohol and potatoes.. A URL-level metric would likely label the second result as more relevant, which would allow a model to form associations beyond simple word-matching (i.e. it could learn that "alcohol" and "vodka" are associated).