Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I recently ran a test on site traffic that was designed to be a 50/50 split. I actually observed a 50.7% to 49.3% split (154,490 vs 150,151 visitors).
How can I test based on the sample size whether this split is no longer random? Is it a test of proportions?
Also, if there is a flaw in the randomisation in this instance, does that invalidate the results of the test?
Thanks for any and all help!
Randomness does not implies a 50/50 probability. You can have randomness as long as the probability is not 100/0 or 0/100. However, I believe it is not the randomness you are really asking here. In stead what you really asking is how to test whether the random process is bias.
Formally, let's use $Z \in (0, 1)$ to denote the outcome of an experiment of assigning a user. Then $X$ follows a Berniulli distribution with some probability $p$. The sum of the a series of Berniulli variables $X = Z_1 + Z_2 + ...+Z_n$ is a Binomal variable. We have also obvsered samples ($n = 304,641$) drawn from the experiment.
Your question is essentially a hypothesis test with $H_0: p=.5, H_a:p\neq.5$.
Given the null hypothesis, the mean and variance of X can be calculated.
$\mu_x = np = 304,641 \times 0.5 = 152,320.5$
$\sigma_x^2 = np(1-p) = 76,160.25$ ($\sigma_x = 275.97$)
Because our $n$ is large, we can approximate the binomal distirbution by a normal distribution $N(\mu_x, \sigma_x )$. It follows, assuming the null hypothesis, then $P(x >= 154,490) \approx 0.0$ . So we can reject the null hypothesis and conclude that the random process is not a 50/50 process.
