Before launching an A/B test, what are the methods to ensure that the population split in control and target group is random for a particular label say, purchase rate.
1
-
$\begingroup$ when you want to apply t-test on two groups that are related(they are generated from a single defined population, how and why you should check for randomness. $\endgroup$– Subhash C. DavarJul 28, 2020 at 12:51
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
Just split them 50/50 (or another percentage, depending on what you want). Once you want to test a specific metric, make sure that if you can only generate that for a portion of the A group because of some specific property (ie you need information C for this metric) to compare it only to the people in group B that also have information C to avoid bias.
Example: I have 200,000 customers. I split them 50/50. I'm testing my new recommendation method against a random advertisement. However my recommender only works if they have searched in the last month. After getting back the results, I will have to check that against the control group filtered to have only people that have searched in the past month. People that have searched in the past month are much more likely to have a stronger kind of connection to you, so if you would check against the whole group you would overestimate the performance of your recommender.
Alternatively you could already split up the groups knowing whether they have information C or not, but you might want to test different things so splitting before hand can give you extra info, at the small cost that your groups might be slightly unequal, but for a big sample size this is not an issue at all.
answered Feb 26, 2016 at 15:58
$\begingroup$
$\endgroup$
3
In some cases it is useful to do stratified sampling. For example, imagine you have 4 different groups in your population each with differing n (e.g., A = 10k, B = 20k, C = 30k D = 40k). In creating your train/test you may want to ensure that each subset maintains the same proportions of subgroup. This can ensure better equivalence of train/test than pure random assignment (which could result in lopsided train/test).
answered Feb 26, 2016 at 16:10
-
$\begingroup$ Actually, I was looking for some methods which can ensure randomization on future data in A/B testing. Say if I am launching an experiment for a offer on a product, and I created two groups, control(A) and target(B). How do I make sure(or maybe, just ensure at best) that B does not consist of all non-buyers (or by random chance, all purchasers go into A). While preparing training/test data I can ensure this since its an offline process and I already have class distributions from each of groups. $\endgroup$– vickFeb 26, 2016 at 18:26
-
2$\begingroup$ It seems like you are asking "how does one ensure comparability between conditions/groups"? You want to ensure that the same type of people are in both your control and test groups. The standard approach is random selection and random assignment to groups. Before running your experiment, you can always do a statistical test to ensure no differences between groups. On a experiment testing a diet drug, you would want to ensure that your test/control groups have the same starting weight, similar age ranges, etc. This gives you an opportunity to re-random assign before your actual experiment. $\endgroup$ Feb 26, 2016 at 19:13
-
$\begingroup$ When you eval a model you feed it existing labelled data (e.g., person did/didn't buy a product) & have the model make its own predictions (for each person, predict the likelihood they will buy product). During eval you just compare how well the model's predictions compare to the "ground truth" established by labelled data. The idea of eval is to get an approx idea of how the model will perform when it is deployed and used to predict in-the-wild data. When the model actually get's deployed, the data in-the-wild is by definition un-labelled... its the job of the model to give this data labels. $\endgroup$ Feb 26, 2016 at 19:16
$\begingroup$
$\endgroup$
You might want to provide some more information here. This is either very straightforward or incredibly difficult, depending on your needs.
@Brandon Loudermilk is dead on for recommending stratification, especially for something like purchase rate, prior product purchased, etc. This will get you a long ways, but there are a few places this is heavily dependent, and a lot of this boils down to experimental design:
- What are you hoping to test
- What defines success? A 10% improvement? A 0.001% improvement?
- How many observations do you expect in each group?
- Are you assigning users to groups on the fly, or are you designing this prior?
If you're doing this on the fly, for instance, if your tool can split over a given variable/quartile of that variable, it should place users into A or B depending on the strata they belong to. That would cause every using coming in to have a 50/50 chance of landing in the right bucket - you should never end up with only high spenders in one bucket.
That being said - if you can design this before the test, you should be able to ensure this split fairly easily by doing this prior to the test. However, you might want to look up blocking/experimental design if you have a complicated experimental setup. As a final note, remember that you cannot test for randomness - only test for signs against randomness.
