I've couple of recommendation algorithms that I want to A/B test. Algorithm A has 90% user coverage and algorithm B has 95% user coverage. That means if the algorithms are asked to provide recommendations for 1000 users, algorithm A can give it for 900 of the users and algorithm B can give it for 950 other users. Say for example out of these 1000 users 87% has recommendations from both algorithm, 3% has recommendations from only algorithm A and 8% has recommendations from only algorithm B. What would be the ideal approach to bucket the users

  1. Bucket only those 87% users as that can generate even bucketing, but the higher coverage of algorithm B won't reflect in A/B test.
  2. Bucket all 95% users, but that could cause uneven bucketing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.