Consider the problem of learning to rank for Google-like searching - i.e., learning to return a good ordering of URL's when given a query. Most (if not all) current evaluation metrics for this problem are URL-based. Current evaluation metrics like Mean Reciprocal Rank (MRR) and Discounted Cumulative Gain (DCG) sum up weighted relevance scores for the pages in a ranked list. If a ranked list contains n pages, for instance, these metrics give a sum of n terms.
From my own literature review, it seems no metrics incorporate words. Although some theoretically incorporate "information nuggets", none incorporate "information nuggets=words" in practice. I have not found recent or past literature that concretely utilizes words, phrases, or anything similar in scoring ranked lists. Since I haven't found past literature, it does not seem that word-based metrics were attempted once and then replaced with coarser URL-based metrics. Documents contain words, as do the snippets returned in a Google-like ranked list, so it seems natural that word-based metrics should have been attempted. Thoughts?
(Note: One obvious objection is that TREC-based assessment of "word relevance" is a vague and even expensive endeavor (asking assessors to mark for words for relevance). But there's other ways to get relevance feedback that ignore this problem, such as using the dwell time on pages as a proxy.)