3
$\begingroup$

Consider a recommender system which sends discount coupons for cakes to visitors on some website. There are 2 cases:

  • good case: when a customer visits the website with no intent of buying a cake, but their behavior changed as a result of the seeing the recommendation. The website gained a new purchase.
  • bad case: when a visitor already had the intention of buying a cake before seeing the recommendation. The website lost money by sending the discount.

How is this problem called? How is it quantified? What are the common solutions?

$\endgroup$

2 Answers 2

6
$\begingroup$

This is a case for A/B testing:

Two versions of the website are prepared, one offering the coupon and another not offering it. Users are randomly split into two groups, some of them getting the coupon version and others the regular version (control group).

The parameters have to be designed in advance of course: how long the experiment lasts, are the two versions simultaneous or happening one after the other, possibly proportion of each case, etc.

At the end of the experiment the effect of the coupons can be measured by doing a statistical test on whether the coupon modifies the proportion of visitors who buy a cake.

$\endgroup$
6
$\begingroup$

As Erwan has pointed out, the straightforward way to tackle this problem is using A/B testing, so I'll point out another technique to solve the issue, which is counterfactual evaluation.

In counterfactual evaluation, you estimate the impact of your RecSys offline. This can be useful if doing an A/B test is expensive or impossible. Also, in A/B testing the experiment cycles are long. Or maybe you are working in a research environment and you don't have access to A/B testing. I leave you here a super-interesting post by Eugene Yan - RecSys expert at Amazon - on this topic.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.