I have two different search engines that always return the same results but in different orders. The results consist of websites along with confidence scores, which range from 100 to 10,000. The maximum number of results is capped at 99. Assume the results of first search engine is ideal.

For example, consider the following sixty results by two different search engines E1 and E2:

E1 results ( site:score) E2 results (site:score)
$w_1$ - 195 $w_4$ - 199
$w_2$ - 192 $w_{24}$ - 192
$w_3$ - 190 $w_{36}$ - 189
$w_4$ - 186 $w_3$ - 166
... ...
$w_{60}$ - 105 $w_6$ - 101

Given this setup, I am looking for a metric commonly used in NLP literature to compare these ranked, weighted results. The intention is to know how similar the second list wrt to the first. The metric should meet the following criteria:

  1. Suitable for lists that are finite, small, and complete, with just websites order changed.

  2. (optional) Includes a variable or mechanism that controls the importance of the top-k results, where k can vary from 1 to n. For instance, if k=1, the metric should primarily focus on the first result, whereas if k=n, it should give equal importance to the entire list.

Note: The websites are permuted and not scores. Scores can be different in two lists, both in order and values. For example: the highest score in E1 results is for w1, which is 95 and in E2, it is 199 for w4.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.