How to calculate precision at K and NDCG for ranking algorithms

Question

I am ranking a filtered item list as per user's metadata and historical behaviour.

Now how to calculate metrices like precision at K?

One approach could be - Divide historical data in training and test dataset(e.g. 90% and 10%). Now pick all the items for a user from test data and randomize the order and pass it to the ranking algorithm.

Now compare actual items for the user from test data to the ranked output.

Answer 1

Precision at k (precision@k) requires labeled data. Each item needs to be labeled as relevant or not relevant for the given user.

$$ precision@k = \frac{Number\ of\ relevant\ @k}{Number\ of\ recommended\ items\ @k} $$

Labeling data is independent of train/test split. Train/test split is useful to avoid overfitting and for estimating a model's ablity to generalize.

Answer 2

examlpe data: $$\begin{array}{|c|c|c|c|c|c|} \hline iterms\_id & predict\_score & predict\_rank & true\_score & true\_rank & pression@k \\ \hline 2 & 0.8 & 1 & 0.7 & 3 & 1 \\ \hline 3 & 0.7 & 2 & 0.6 & 4 & 1 \\ \hline 4 & 0.6 & 3 & 0.9 & 1 & 1 \\ \hline 1 & 0.4 & 4 & 0.8 & 2 & 0.75 \\ \hline 5 & 0.2 & 5 & 0.1 & 5 & 0.8 \\ \hline \end{array}$$

$ DCG_k = \sum_{i=1}^{k}\frac{2^{rel_i} - 1}{log_2{(i + 1)}} $

$ IDCG_k = \sum_{i=1}^{k}\frac{2^{rue_i} - 1}{log_2{(i + 1)}} $

$ NDCG_k = \frac{DCG_k}{IDCG_k} $

$ DCG_5 = \frac{2^{0.7} -1}{log_2(1+1)} + \frac{2^{0.6} -1}{log_2(2+1)} + \frac{2^{0.9} -1}{log_2(3+1)} + \frac{2^{0.8} -1}{log_2(4+1)} + \frac{2^{0.1} -1}{log_2(5+1)} = 1.72986 $

$ IDCG_5 = \frac{2^{0.9} -1}{log_2(1+1)} + \frac{2^{0.8} -1}{log_2(2+1)} + \frac{2^{0.7} -1}{log_2(3+1)} + \frac{2^{0.6} -1}{log_2(4+1)} + \frac{2^{0.1} -1}{log_2(5+1)} = 1.89577 $

$NDCG_5 = \frac{1.72986}{1.89577} = 0.91248 $

hope this helps.QTQ