2
$\begingroup$

Consider a learning to rank setting, where I'm learning from N items displayed to the user for every user query. Suppose I can quantify the probability of examination $P[E_i]$ of each position $i$ given that the first position was examined (i.e. relative to the first position). The user feedback is binary - click / no-click.

Now suppose I'm using the softmax-crossentropy listwise loss (don't ask why - suppose it's a constraint in the system) on each query. What would be the 'correct' way to incorporate the position bias information?

Intuitively, a click on a lower position is more informative than a click on a higher position. So it would be beneficial to weigh the loss inversely proportional to $P[E_i]$ for a click on position $i$. On the other hand, a non-click on a low position is less informative, so it would make sense to weigh the terms inside the soft-max in such a way that lower examination probability results in a smaller effect. But what are the correct weights? It's not obvious.

$\endgroup$

1 Answer 1

0
$\begingroup$

One way to incorporate the information from position bias is discounted cumulative gain (DCG). DCG sums the usefulness, or gain, of the results discounted by their position in the result list. Typically normalized DCG (NDCG) is used, which normalizes across queries the cumulative gain at each position for a particular rank position.

NDCG is commonly used as an evaluation metric. SoftRank is a version that has been adapted to be used as a loss function. There are also more contemporary versions such as Weighted Approximate Pairwise Ranking Loss (WARPLoss).

$\endgroup$
2
  • $\begingroup$ Yes, I know. But NDCG is mainly useful for evaluation - it's not a loss I that is easy to differentiate through and learn a model. $\endgroup$ yesterday
  • $\begingroup$ Good point, I revised my answer to be more precise. $\endgroup$ yesterday

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.