Sparse IR with user feedback

Question

I'm considering a problem framing within an information retrieval context.

I have a sequence of documents that feature different attributes. In the web context, these would be webpages. One attribute could be "is this a top-10 content creator," etc. When we convert multi-labeled values to binary indicators, we end up with a matrix like:

   a b c d e f
A: 1 0 0 0 1 0
B: 0 1 0 0 1 0
C: 0 0 1 0 0 0
D: 0 1 0 0 0 1
E: 0 0 0 0 1 0
F: 0 0 1 0 0 1

We can ask a user a series of progressive questions about their preferences within this dataset. Eg. "Do you care about it being from a top 10 content creator?"

However, these are just preferences. Just because we know the "answer" to a question doesn't immediately invalidate documents with that attribute. It should just bump it down in the eventual ranking.

The task at hand becomes knowing A) which questions to ask to maximize information gain and B) ranking the resulting list. This seems like it would be an existing research area but thus far I haven't been able to find anything on it. Is there a name for this area of algorithm design?

Answer 1

This is a learning to rank problem, in particular feature selection for learning to rank.

There are many ways to solve the problem. One common method is maximizing feature importance while minimizing similarity. In other words, rank documents based on the most valuable and unique features. This is a variation of Minimum-redundancy-maximum-relevance (mRMR) feature selection.

To maximize feature importance, first rank all documents using all features, evaluate performance with evaluation measures, and show documents with the highest importance scores.

To minimize similarity, find features that have the least redundancy. Redundancy could be measured as the average value of all mutual information values between each feature pair.

"Feature Selection for Ranking" goes into greater detail.

Once the features are ordered with mRMR, the user can select specific features. Then re-rank documents based on the user-selected features.