Cross posting this from Cross Validated:
I've seen this question asked before, but I have yet to come across a definitive source answering the specific questions:
- What's the most appropriate statistical test to apply to a small A/B test?
- What's the R code and interpretation to analyze a small A/B test?
I'm running a small test to figure out which ads perform better. I have the following results:
I think it's safe to say these numbers are small and likely to be not normally distributed. Also, it's click data so there's a binary outcome of clicked or not and the trials are independent.
In analyzing each position for significance, I think comparison with a binomial or Poisson distribution makes the most sense.
According to the OpenIntro Stats (and other sources) book, a variable follows a Poisson distribution "... if the event being considered is rare, the population is large, and the events occur independently of each other."
The same source classifies a binomial variable approximately the same way adding that the probability of success is the same and the number of trials is fixed.
I appreciate this is not an either/or decision and analysis can be done using both distributions.
Given A/B (split) testing is a science that has been practiced for several years, I imagine that there is a canonical test. However, looking around the internet, I mostly come across analysis that uses the standard normal distribution. That just seems wrong :)
Is there a canonical test to use for A/B tests with small #'s of clicks?
Interpretation and R code
I've used the following R code to test significance for each position:
binom.test(7, 767, p=(26/753)) Exact binomial test data: 7 and 767 number of successes = 7, number of trials = 767, p-value = 1.077e-05 alternative hypothesis: true probability of success is not equal to 0.03452855 95 percent confidence interval: 0.003676962 0.018713125 sample estimates: probability of success 0.009126467
I interpret this result to mean: The probability of success in the test group is indeed different than the control group with a 95% confidence interval that the success probability is between .368% and 1.87%
ppois(((26-1)/753), lambda=(7/767), lower.tail = F)  0.009084947
I interpret this result to mean: Given a Poisson distribution with a click rate of 7 per 767 trials, there is a 0.9% chance of having a click rate of 26 or more per 753 trials in the same distribution. Contextualized in the ad example, there is a .1% chance that the control ad actually performs the same as the test ad.
Is the above interpretation correct? Does the test and interpretation change with the different positions (i.e. are the results of the Poisson test more appropriate for Position 3 given the small numbers)?