I successfully extract keywords from documents in my corpus in a variety of different ways (like running pagerank on an cooccurance matrix, or textrank, or using a similarity matrix, or generating my matrix from only certain parts of speech, etc). I'm currently stuck figuring out how to determine which of the methods is best. I know that keywords have no objectively correct answer. I'm not trying to get an objective answer as to whats right, so much as what the people I'm doing the extraction for prefer.
The people who will be using the keywords have given me a benchmark dataset.. sort of. They had a bunch of people go through a subset of my corpus and identify what they thought were the correct keywords. Naturally, not everyone's answers were the same (they did it independently). I'm thinking of just combining everyone's answers and claiming they form the benchmark set. This then raises the question of how I evaluate my algorithms with respect to this benchmark set, which is what I'm looking for advice on.
For one, is there a nice objective way to do this out there already? Every search I find comes up with web search based keywords, but I have no data on search statistics, only my benchmark set. If there is a nice way to do this, how do I include uncertainty? Ie, since my benchmark is a combination of a few people's answers, and no person is all that accurate, why should I expect my algorithms to be? Also, how does one evaluate whether an answer is correct or not if the words are similar but not exact synonyms (like my benchmark includes the word exercise and an algorithm submits the word work)?