Testing whether observed control/test split invalidates my assumption of 50/50 randomised traffic

I recently ran a test on site traffic that was designed to be a 50/50 split. I actually observed a 50.7% to 49.3% split (154,490 vs 150,151 visitors).

How can I test based on the sample size whether this split is no longer random? Is it a test of proportions?

Also, if there is a flaw in the randomisation in this instance, does that invalidate the results of the test?

Thanks for any and all help!

• Please explain what do you mean by "designed to be a 50/50 split". If the traffic is being diverged at random, then why do you think it is not random? Nov 20, 2017 at 13:09
• The traffic is being split externally by a partner website who is then directing the traffic to our site. As they control the intended 50/50 splitting of traffic I wanted to make sure what we actually observed (51/49) doesn't invalidate this assumption. Nov 20, 2017 at 14:25

Randomness does not implies a 50/50 probability. You can have randomness as long as the probability is not 100/0 or 0/100. However, I believe it is not the randomness you are really asking here. In stead what you really asking is how to test whether the random process is bias.

Formally, let's use $Z \in (0, 1)$ to denote the outcome of an experiment of assigning a user. Then $X$ follows a Berniulli distribution with some probability $p$. The sum of the a series of Berniulli variables $X = Z_1 + Z_2 + ...+Z_n$ is a Binomal variable. We have also obvsered samples ($n = 304,641$) drawn from the experiment.

Your question is essentially a hypothesis test with $H_0: p=.5, H_a:p\neq.5$.

Given the null hypothesis, the mean and variance of X can be calculated.

$\mu_x = np = 304,641 \times 0.5 = 152,320.5$

$\sigma_x^2 = np(1-p) = 76,160.25$ ($\sigma_x = 275.97$)

Because our $n$ is large, we can approximate the binomal distirbution by a normal distribution $N(\mu_x, \sigma_x )$. It follows, assuming the null hypothesis, then $P(x >= 154,490) \approx 0.0$ . So we can reject the null hypothesis and conclude that the random process is not a 50/50 process.