I saw an article about an A/B test that google had performed way back. They wanted to decide what shade of blue a button should be and how that affects click-through rate. They divided users randomly into 100 buckets - each corresponding to a shade of blue they wanted to check (so the color is a factor with 100 levels).

Now this is all well and good if all the buckets (or "treatment groups") sufficiently represent the target population. In other words, google had the liberty of taking large enough random samples from the population they wanted to target.

My question is: how would the test/sampling/bucketing/analysis have to be modified if there is a constraint on the number of replications?

For those experienced in experiment design / AB testing, I would be extremely grateful if you could include some other practical problems or constraints that arise in A/B testing - the kind that an average firm faces.

  • $\begingroup$ In the first case there is no need to do A/B testing, right? Since the target users are simply not there. Instead one can transfer the tests if possible to other areas where indeed the target group exists (eg another site) and adapt the results from there. $\endgroup$
    – Nikos M.
    Jun 19, 2021 at 12:24
  • $\begingroup$ @NikosM.: You're right. I will edit the question to only include the second case. Thanks! $\endgroup$ Jun 19, 2021 at 12:27


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.