# How to calculate precision at K and NDCG for ranking algorithms

I am ranking a filtered item list as per user's metadata and historical behaviour.

Now how to calculate metrices like precision at K?

One approach could be - Divide historical data in training and test dataset(e.g. 90% and 10%). Now pick all the items for a user from test data and randomize the order and pass it to the ranking algorithm.

Now compare actual items for the user from test data to the ranked output.

Precision at k (precision@k) requires labeled data. Each item needs to be labeled as relevant or not relevant for the given user.

$$precision@k = \frac{Number\ of\ relevant\ @k}{Number\ of\ recommended\ items\ @k}$$

Labeling data is independent of train/test split. Train/test split is useful to avoid overfitting and for estimating a model's ablity to generalize.

• Brian, I understand precision at K. My question is around how to prepare test data for ranking algorithms. I added one approach in the question. I wanted to see how other people are doing it
– Dev
Jan 14, 2020 at 4:20

examlpe data: $$\begin{array}{|c|c|c|c|c|c|} \hline iterms\_id & predict\_score & predict\_rank & true\_score & true\_rank & pression@k \\ \hline 2 & 0.8 & 1 & 0.7 & 3 & 1 \\ \hline 3 & 0.7 & 2 & 0.6 & 4 & 1 \\ \hline 4 & 0.6 & 3 & 0.9 & 1 & 1 \\ \hline 1 & 0.4 & 4 & 0.8 & 2 & 0.75 \\ \hline 5 & 0.2 & 5 & 0.1 & 5 & 0.8 \\ \hline \end{array}$$

$$DCG_k = \sum_{i=1}^{k}\frac{2^{rel_i} - 1}{log_2{(i + 1)}}$$

$$IDCG_k = \sum_{i=1}^{k}\frac{2^{rue_i} - 1}{log_2{(i + 1)}}$$

$$NDCG_k = \frac{DCG_k}{IDCG_k}$$

$$DCG_5 = \frac{2^{0.7} -1}{log_2(1+1)} + \frac{2^{0.6} -1}{log_2(2+1)} + \frac{2^{0.9} -1}{log_2(3+1)} + \frac{2^{0.8} -1}{log_2(4+1)} + \frac{2^{0.1} -1}{log_2(5+1)} = 1.72986$$

$$IDCG_5 = \frac{2^{0.9} -1}{log_2(1+1)} + \frac{2^{0.8} -1}{log_2(2+1)} + \frac{2^{0.7} -1}{log_2(3+1)} + \frac{2^{0.6} -1}{log_2(4+1)} + \frac{2^{0.1} -1}{log_2(5+1)} = 1.89577$$

$$NDCG_5 = \frac{1.72986}{1.89577} = 0.91248$$

hope this helps.QTQ