What are some alternative evaluation metrics for ranking problems, that could help when evaluation is done on heavily biased data?

Example - if we sort items by their price and want to evaluate alternative sorting, all the relevant items in the data will be with cheap price, due to people choosing one of the top results. Ideally a metric will discount a relevant example’s score that was up in the original ranking.

Standard metrics like MRR, (N)DCG@k, MAP@k and similar don’t work like that.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.