1
$\begingroup$

I am working in a multilabel recommender project and I try to evaluate it as a ranking problem.

I calculate recall@k and precision@k which both looks quite well. Recall increases and Precision decreases as I try higher K values, which is expected.

However, the NDCG@K increases up to a certain K and after that it stays the same. How can we explain such a behaviour?

$\endgroup$

1 Answer 1

0
$\begingroup$

It is hard to give an appropriate answer regarding your specific problem (mainly the value of K around which you don't see an increase anymore), and I guess you might have moved on from this question.

But from a theoretical perspective the DCG is the ratio of relevance divided by a discount function. On a multilabel ranking problem you'll use a binary relevance function (either 0 or 1, depending if the label belongs to the ground truth label set). The discount function is by definition a decreasing function, so for large values of K, the contributions of ill ranked will vanish to 0. The most common version of DCG uses a $\frac{1}{\log(rank)}$ discount function, thus contributions of very ill ranked positive labels will end up being of $O(\frac{1}{\log K})$, so basically 0 for very large K.

This can be alleviated by changing your discount function for a function that will decrease slower (e.g by changing the base of your log, or using log(log(rank), a low power of rank).

Also in general NDCG@K converges to 1 when K goes to infinity.

See this paper for some theoretical work characterizing NDCG.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.