2
$\begingroup$

I have three questions:

  1. How can we assess (or measure) the performance of the ranking algorithms?

  2. Are there any specific measures, or performance metrics, for this?

  3. More specifically, how can we compare the performance of AHP-based ranking and probabilistic ranking algorithms?

Improve this question
$\endgroup$
4
  • $\begingroup$ poke around contemporary information retrieval research for ranking metrics. $\endgroup$
    – David Marx
    Jan 4, 2018 at 13:40
  • $\begingroup$ I have gone through, but could not find specific answers, especially for the third question $\endgroup$
    – mani
    Jan 4, 2018 at 13:57
  • $\begingroup$ I've never seen AHP in the machine learning literature. What is the context? $\endgroup$
    – Emre
    Jan 4, 2018 at 17:54
  • $\begingroup$ The context is formulation of web service selection (ranking) problem using multicriteria decision making process (Analytic hierarchy process). I am not sure the response of @Tomaso Neri is relevant or not? We actually want to compare deterministic AHP-based ranking method with that of probabilistic ranking algorithm. $\endgroup$
    – mani
    Jan 5, 2018 at 12:06

1 Answer 1

Reset to default
1
$\begingroup$

Kaggle's famous competition Chess ratings - Elo versus the Rest of the World, that aimed "to discover whether other approaches can predict the outcome of chess games more accurately than the workhorse Elo rating system", used this structure

Competitors train their rating systems using a training dataset of over 65,000 recent results for 8,631 top players. Participants then use their method to predict the outcome of a further 7,809 games

A similar structure - starting from a complete dataset, using first part for training and last part to check the outcome - could be useful to measure the performance of the ranking algorithms.

Improve this answer
$\endgroup$
1
  • $\begingroup$ The context is formulation of web service selection (ranking) problem using multicriteria decision making process (Analytic hierarchy process). I am not sure how machine learning is relevant here? We actually want to compare deterministic AHP-based ranking method with that of probabilistic ranking algorithm! $\endgroup$
    – mani
    Jan 5, 2018 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.