Combining multiple ranked lists

Question

Suppose I'm given two ranked lists, A and B, with each item in the lists being associated with a score:

A = [(I_2, 6), (I_4, 5), (I_1, 3), (I_5, 1)] - the scores are in descending order
B = [(I_4, 1), (I_3, 0.5), (I_7, 0.25), (I_8, 0.1)] - the scores are in descending order

I want to check whether a combination of the lists produces better results than any individual list. One way to combine A and B is to average over the ranks, sort the tuples and return the first 4. However, I'm wondering if there is a way to utilize the scores for a combination. The thing is, the lists A and B are coming from different models, so a comparison of the scores is only meaningful between the items of the same list (model). How could one proceed, taking this into consideration? By normalizing the scores (e.g., make 1 a maximum of both the lists)? If so, how would you normalize it? Also, beside the averaging and picking the ones corresponding to highest scores, are there other ways to combine the lists?

Answer 1

Your principle complaint is that scores are not comparable outside of the model that predicted them. That is, they lack common units.

Well, let's put them on the same page, let's use common units. There are at least two approaches.

(1.) business value

Suppose that each item (e.g. "I_4") is a business policy, such as "Use creative I_4 when advertising to the target demographic," and then we measure business utility, perhaps revenue or CTR click through rate.

Well, now it's straightforward. Choose a utility cutoff, which reflects the cost of running a trial for a low-performing policy. Some trials simply won't be worth running. Run trials with each of the most promising policies in both lists, and record each measurement of utility. Now combine the lists.

Rank ordering within each list simply helped us with the cutoff decisions. An external trial let us map score to common unit.

(2.) calibration

Suppose those scores relate to classification performance. Then Platt scaling and other calibration methods can find a monotonic mapping from raw score to a proper probability.