Pairwise comparison models such as that of Bradley and Terry can easily be extended to your case, when you have pairwise comparison probabilities instead of binary outcomes.
Let $N$ be the number of items, and let $p_{ij}$ be the probability that query $j$ is better than query $i$.
Then, the log-likelihood of the Bradley-Terry parameters $\lambda_1, \ldots, \lambda_N$ given probabilities $\{ p_{ij} \}$ is
$$
\sum_{i,j} p_{ij} [\log(\lambda_j) - \log(\lambda_i + \lambda_j)]
$$
This can be reparametrized into a convex function, and the maximum-likelihood parameters can be found by one of many convex opimization methods.
Here is a simple Python algorithm that will find the ML estimate, using a minorization-maximization approach.
import numpy as np
def mle(pmat, max_iter=100):
n = pmat.shape[0]
wins = np.sum(pmat, axis=0)
params = np.ones(n, dtype=float)
for _ in range(max_iter):
tiled = np.tile(params, (n, 1))
combined = 1.0 / (tiled + tiled.T)
np.fill_diagonal(combined, 0)
nxt = wins / np.sum(combined, axis=0)
nxt = nxt / np.mean(nxt)
if np.linalg.norm(nxt - params, ord=np.inf) < 1e-6:
return nxt
params = nxt
raise RuntimeError('did not converge')
Example usage:
import itertools
# Generating pairwise probability matrix.
pmat = np.zeros((10, 10))
for i, j in itertools.permutations(range(10), r=2):
pmat[i][j] = (j + 1) / (i + j + 2)
# Estimating Bradley-Terry model parameters.
params = mle(pmat)
# Ranking (worst to best).
ranking = np.argsort(params)
Source: I am an author of a Python library for parameter inference in various statistical comparison models, choix.
