I recently ran a test on site traffic that was designed to be a 50/50 split. I actually observed a 50.7% to 49.3% split (154,490 vs 150,151 visitors).

How can I test based on the sample size whether this split is no longer random? Is it a test of proportions?

Also, if there is a flaw in the randomisation in this instance, does that invalidate the results of the test?

Thanks for any and all help!

  • $\begingroup$ Please explain what do you mean by "designed to be a 50/50 split". If the traffic is being diverged at random, then why do you think it is not random? $\endgroup$
    – Louis T
    Nov 20, 2017 at 13:09
  • $\begingroup$ The traffic is being split externally by a partner website who is then directing the traffic to our site. As they control the intended 50/50 splitting of traffic I wanted to make sure what we actually observed (51/49) doesn't invalidate this assumption. $\endgroup$ Nov 20, 2017 at 14:25

1 Answer 1


Randomness does not implies a 50/50 probability. You can have randomness as long as the probability is not 100/0 or 0/100. However, I believe it is not the randomness you are really asking here. In stead what you really asking is how to test whether the random process is bias.

Formally, let's use $Z \in (0, 1)$ to denote the outcome of an experiment of assigning a user. Then $X$ follows a Berniulli distribution with some probability $p$. The sum of the a series of Berniulli variables $X = Z_1 + Z_2 + ...+Z_n$ is a Binomal variable. We have also obvsered samples ($n = 304,641$) drawn from the experiment.

Your question is essentially a hypothesis test with $H_0: p=.5, H_a:p\neq.5$.

Given the null hypothesis, the mean and variance of X can be calculated.

$\mu_x = np = 304,641 \times 0.5 = 152,320.5$

$\sigma_x^2 = np(1-p) = 76,160.25$ ($\sigma_x = 275.97$)

Because our $n$ is large, we can approximate the binomal distirbution by a normal distribution $N(\mu_x, \sigma_x )$. It follows, assuming the null hypothesis, then $P(x >= 154,490) \approx 0.0$ . So we can reject the null hypothesis and conclude that the random process is not a 50/50 process.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.