2
$\begingroup$

I have three questions:

  1. How can we assess (or measure) the performance of the ranking algorithms?

  2. Are there any specific measures, or performance metrics, for this?

  3. More specifically, how can we compare the performance of AHP-based ranking and probabilistic ranking algorithms?

$\endgroup$
4
  • $\begingroup$ poke around contemporary information retrieval research for ranking metrics. $\endgroup$ – David Marx Jan 4 '18 at 13:40
  • $\begingroup$ I have gone through, but could not find specific answers, especially for the third question $\endgroup$ – mani Jan 4 '18 at 13:57
  • $\begingroup$ I've never seen AHP in the machine learning literature. What is the context? $\endgroup$ – Emre Jan 4 '18 at 17:54
  • $\begingroup$ The context is formulation of web service selection (ranking) problem using multicriteria decision making process (Analytic hierarchy process). I am not sure the response of @Tomaso Neri is relevant or not? We actually want to compare deterministic AHP-based ranking method with that of probabilistic ranking algorithm. $\endgroup$ – mani Jan 5 '18 at 12:06
1
$\begingroup$

Kaggle's famous competition Chess ratings - Elo versus the Rest of the World, that aimed "to discover whether other approaches can predict the outcome of chess games more accurately than the workhorse Elo rating system", used this structure

Competitors train their rating systems using a training dataset of over 65,000 recent results for 8,631 top players. Participants then use their method to predict the outcome of a further 7,809 games

A similar structure - starting from a complete dataset, using first part for training and last part to check the outcome - could be useful to measure the performance of the ranking algorithms.

$\endgroup$
1
  • $\begingroup$ The context is formulation of web service selection (ranking) problem using multicriteria decision making process (Analytic hierarchy process). I am not sure how machine learning is relevant here? We actually want to compare deterministic AHP-based ranking method with that of probabilistic ranking algorithm! $\endgroup$ – mani Jan 5 '18 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.