I've couple of recommendation algorithms that I want to A/B test. Algorithm A has 90% user coverage and algorithm B has 95% user coverage. That means if the algorithms are asked to provide recommendations for 1000 users, algorithm A can give it for 900 of the users and algorithm B can give it for 950 other users. Say for example out of these 1000 users 87% has recommendations from both algorithm, 3% has recommendations from only algorithm A and 8% has recommendations from only algorithm B. What would be the ideal approach to bucket the users
- Bucket only those 87% users as that can generate even bucketing, but the higher coverage of algorithm B won't reflect in A/B test.
- Bucket all 95% users, but that could cause uneven bucketing.