2
$\begingroup$

I am ranking a filtered item list as per user's metadata and historical behaviour.

Now how to calculate metrices like precision at K?

One approach could be - Divide historical data in training and test dataset(e.g. 90% and 10%). Now pick all the items for a user from test data and randomize the order and pass it to the ranking algorithm.

Now compare actual items for the user from test data to the ranked output.

$\endgroup$

2 Answers 2

1
$\begingroup$

Precision at k (precision@k) requires labeled data. Each item needs to be labeled as relevant or not relevant for the given user.

$$ precision@k = \frac{Number\ of\ relevant\ @k}{Number\ of\ recommended\ items\ @k} $$

Labeling data is independent of train/test split. Train/test split is useful to avoid overfitting and for estimating a model's ablity to generalize.

$\endgroup$
1
  • $\begingroup$ Brian, I understand precision at K. My question is around how to prepare test data for ranking algorithms. I added one approach in the question. I wanted to see how other people are doing it $\endgroup$
    – Dev
    Jan 14, 2020 at 4:20
1
$\begingroup$

examlpe data: $$\begin{array}{|c|c|c|c|c|c|} \hline iterms\_id & predict\_score & predict\_rank & true\_score & true\_rank & pression@k \\ \hline 2 & 0.8 & 1 & 0.7 & 3 & 1 \\ \hline 3 & 0.7 & 2 & 0.6 & 4 & 1 \\ \hline 4 & 0.6 & 3 & 0.9 & 1 & 1 \\ \hline 1 & 0.4 & 4 & 0.8 & 2 & 0.75 \\ \hline 5 & 0.2 & 5 & 0.1 & 5 & 0.8 \\ \hline \end{array}$$

$ DCG_k = \sum_{i=1}^{k}\frac{2^{rel_i} - 1}{log_2{(i + 1)}} $

$ IDCG_k = \sum_{i=1}^{k}\frac{2^{rue_i} - 1}{log_2{(i + 1)}} $

$ NDCG_k = \frac{DCG_k}{IDCG_k} $

$ DCG_5 = \frac{2^{0.7} -1}{log_2(1+1)} + \frac{2^{0.6} -1}{log_2(2+1)} + \frac{2^{0.9} -1}{log_2(3+1)} + \frac{2^{0.8} -1}{log_2(4+1)} + \frac{2^{0.1} -1}{log_2(5+1)} = 1.72986 $

$ IDCG_5 = \frac{2^{0.9} -1}{log_2(1+1)} + \frac{2^{0.8} -1}{log_2(2+1)} + \frac{2^{0.7} -1}{log_2(3+1)} + \frac{2^{0.6} -1}{log_2(4+1)} + \frac{2^{0.1} -1}{log_2(5+1)} = 1.89577 $

$NDCG_5 = \frac{1.72986}{1.89577} = 0.91248 $

hope this helps.QTQ

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.