Cross posting this from Cross Validated:

I've seen this question asked before, but I have yet to come across a definitive source answering the specific questions:

  • What's the most appropriate statistical test to apply to a small A/B test?
  • What's the R code and interpretation to analyze a small A/B test?

I'm running a small test to figure out which ads perform better. I have the following results:

Position 1:

variation,impressions,clicks row-1,753,26 row-3,767 7

Position 2:

variation,impressions,clicks row-1,753,16 row-3,767 13

Position 3:

variation,impressions,clicks row-1,753,2 row-3,767 7

I think it's safe to say these numbers are small and likely to be not normally distributed. Also, it's click data so there's a binary outcome of clicked or not and the trials are independent.

Appropriate test

In analyzing each position for significance, I think comparison with a binomial or Poisson distribution makes the most sense.

According to the OpenIntro Stats (and other sources) book, a variable follows a Poisson distribution "... if the event being considered is rare, the population is large, and the events occur independently of each other."

The same source classifies a binomial variable approximately the same way adding that the probability of success is the same and the number of trials is fixed.

I appreciate this is not an either/or decision and analysis can be done using both distributions.

Given A/B (split) testing is a science that has been practiced for several years, I imagine that there is a canonical test. However, looking around the internet, I mostly come across analysis that uses the standard normal distribution. That just seems wrong :)

Is there a canonical test to use for A/B tests with small #'s of clicks?

Interpretation and R code

I've used the following R code to test significance for each position:

Position 1:

binom.test(7, 767, p=(26/753))

Exact binomial test

data:  7 and 767
number of successes = 7, number of trials = 767, p-value = 1.077e-05
alternative hypothesis: true probability of success is not equal to 0.03452855
95 percent confidence interval:
 0.003676962 0.018713125
sample estimates:
probability of success 

I interpret this result to mean: The probability of success in the test group is indeed different than the control group with a 95% confidence interval that the success probability is between .368% and 1.87%

ppois(((26-1)/753), lambda=(7/767), lower.tail = F)
[1] 0.009084947

I interpret this result to mean: Given a Poisson distribution with a click rate of 7 per 767 trials, there is a 0.9% chance of having a click rate of 26 or more per 753 trials in the same distribution. Contextualized in the ad example, there is a .1% chance that the control ad actually performs the same as the test ad.

Is the above interpretation correct? Does the test and interpretation change with the different positions (i.e. are the results of the Poisson test more appropriate for Position 3 given the small numbers)?


OK, so here's your data.

dd <- data.frame(position=rep(1:3, each=2), 
                 variation=rep(c(1,3), 3), 
                 impressions=rep(c(753, 767), 3), 

which is

  position variation impressions clicks
1        1         1         753     26
2        1         3         767      7
3        2         1         753     16
4        2         3         767     13
5        3         1         753      2
6        3         3         767      7

The two model assumptions you're thinking about are Binomial

mod.bin <- glm(cbind(clicks, impressions-clicks) ~ variation + position,
               family=binomial, data=dd)

where the dependent variable is constructed to have the count of the event of interest in the first column, and the Poisson

md.pois <- glm(clicks ~ variation + position + offset(log(impressions)), 
               family=poisson, data=dd)

where the log(impressions) offset is necessary whenever the number of trials differs across observations. This means coefficients are interpretable in terms of change in rate not change in count, which is what you want.

The first model generalises the binom.test to a setting with covariates, which is what you have. That gets you a more direct answer to your question, and better (if not perfect) measurement of the relevant uncertainty.


Both models assume no interaction between variation and position ('independent effects'). This may or may not be reasonable. You'd want more replications to investigate that properly. Swap the + for a * to do so.

In this data summary confirms that the two models give rather similar results, so concerns about Poisson vs Binomial don't seem to matter much.

In the wild, count data is usually overdispersed, that is: more variable than you'd expect from a Poisson with a constant rate or a Binomial with constant click probability, often due to unmodeled determinants of click rate / probability. If that's the case then prediction intervals from these models will be too narrow.


The correct model is binomial, both poisson and normal are just approximations. The binomial pdf is defined on the integers between zero and number of trials. The poisson is defined on the integers between 0 and infinity, and normal is on all real variables between +/- infinity.

In other words: for a poisson there is a (possibly small) but non zero probability of having more clicks than impressions. For gaussian you can have even negative clicks. Of course, the particular parameters determine how big an impact this has... probably worth plotting the respective pdfs


The approximation Binomial(k,n,p) ~= Poisson(k,s) (where s = n*p) can be shown under the assumptions:
1) n >> k (to say that n!/(n-k)! ~= n^k),
2) p <<1 (to say that (1-p)^(n-k) ~= (1-p)^n).
It's up to you whether those are sufficiently satisfied. If the exact calculation can be done quickly, in my opinion, it's nice to stay with that.

Also since, if the probability of row 3 sample is different from the row 1 sample, it would almost certainly be on the lower side. It would probably best for you to use
binom.test(7, 767, p=(26/753), alternative='less') the final option indicating that the alternative to your null hypothesis is that the probability is less than 26/753, not equal to. Of course, that's simply just the sum of Binomial probabilities from 0 to 7 (you can check yourself), the interpretation being that this is the probability of having gotten at most 7 rolls from random chance, if the probability truly was 26/753.

Keep in mind the interpretation of that last sentence. These kinds of tests are generally used when we know what the inherent probability is that we're comparing to (e.g. to see if the set of coin flips has a probability significantly different from 1/2 which is what we expect from a fair coin). In this case, we don't know what the probability is that we're comparing to, we're just making the very crude guess that the 26/753 outcome of row 1 reflects the true probability. It's better than a regular Normal t-test in this case, but don't put too much stock in it unless you have a much higher sample size for row 1.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.