Skip to main content
Data Science

Return to Answer

edited body
Source Link
Benoit Descamps
Benoit Descamps
  • 341
  • 2
  • 6

Offline evaluation is very tricky to due to all kind of bias. The most prominent type of bias is position bias. I recommend the following paper (https://arxiv.org/pdf/1608.04468.pdf), which contains metrics I have used myself for monitoring and developement of recommendors for a large sport fashion company. The idea is to apply a counter-factual approach to unbias your estimators with the inverse propensity of the documents.

For example, take the sum of the ranks of the relevant results: enter image description here

As you are using implicity feedback for the relance r_i, it is expected that items y that were ranked higher for some query x. There is therefore an implicit probability of observation of an item given position. This probability can be used to reweight the metric as follows:

enter image description here

It can be shown that this estimator is unbiased as well. The same trick can be used for any metric you wish, NDCG, MAP etc... You can also apply for counter-factual estimation of Key Performance Indices, such as expected conversion, click-through rate, add to cart etc...

Unfortunately, one of the issues with such estimators is that they are known to have a large variance due to the weighting factor. I recommend this paper for explanation (https://arxiv.org/pdf/1801.07030.pdf)

Offline evaluation is very tricky to due all kind of bias. The most prominent type of bias is position bias. I recommend the following paper (https://arxiv.org/pdf/1608.04468.pdf), which contains metrics I have used myself for monitoring and developement of recommendors for a large sport fashion company. The idea is to apply a counter-factual approach to unbias your estimators with the inverse propensity of the documents.

For example, take the sum of the ranks of the relevant results: enter image description here

As you are using implicity feedback for the relance r_i, it is expected that items y that were ranked higher for some query x. There is therefore an implicit probability of observation of an item given position. This probability can be used to reweight the metric as follows:

enter image description here

It can be shown that this estimator is unbiased as well. The same trick can be used for any metric you wish, NDCG, MAP etc... You can also apply for counter-factual estimation of Key Performance Indices, such as expected conversion, click-through rate, add to cart etc...

Unfortunately, one of the issues with such estimators is that they are known to have a large variance due to the weighting factor. I recommend this paper for explanation (https://arxiv.org/pdf/1801.07030.pdf)

Offline evaluation is very tricky due to all kind of bias. The most prominent type of bias is position bias. I recommend the following paper (https://arxiv.org/pdf/1608.04468.pdf), which contains metrics I have used myself for monitoring and developement of recommendors for a large sport fashion company. The idea is to apply a counter-factual approach to unbias your estimators with the inverse propensity of the documents.

For example, take the sum of the ranks of the relevant results: enter image description here

As you are using implicity feedback for the relance r_i, it is expected that items y that were ranked higher for some query x. There is therefore an implicit probability of observation of an item given position. This probability can be used to reweight the metric as follows:

enter image description here

It can be shown that this estimator is unbiased as well. The same trick can be used for any metric you wish, NDCG, MAP etc... You can also apply for counter-factual estimation of Key Performance Indices, such as expected conversion, click-through rate, add to cart etc...

Unfortunately, one of the issues with such estimators is that they are known to have a large variance due to the weighting factor. I recommend this paper for explanation (https://arxiv.org/pdf/1801.07030.pdf)

Source Link
Benoit Descamps
Benoit Descamps
  • 341
  • 2
  • 6

Offline evaluation is very tricky to due all kind of bias. The most prominent type of bias is position bias. I recommend the following paper (https://arxiv.org/pdf/1608.04468.pdf), which contains metrics I have used myself for monitoring and developement of recommendors for a large sport fashion company. The idea is to apply a counter-factual approach to unbias your estimators with the inverse propensity of the documents.

For example, take the sum of the ranks of the relevant results: enter image description here

As you are using implicity feedback for the relance r_i, it is expected that items y that were ranked higher for some query x. There is therefore an implicit probability of observation of an item given position. This probability can be used to reweight the metric as follows:

enter image description here

It can be shown that this estimator is unbiased as well. The same trick can be used for any metric you wish, NDCG, MAP etc... You can also apply for counter-factual estimation of Key Performance Indices, such as expected conversion, click-through rate, add to cart etc...

Unfortunately, one of the issues with such estimators is that they are known to have a large variance due to the weighting factor. I recommend this paper for explanation (https://arxiv.org/pdf/1801.07030.pdf)