When performing an A/B test for the number of clicks for users viewing (each view is an impression) two variants of an ad, a binomial distribution can be assumed where each variant has a constant click-through rate.
Example: Two Ads,
-> Ad one has 1000 impressions and 20 clicks, CTR is 2%;
-> Ad two has 900 impressions and 30 clicks, CTR is 3.3%.
Test whether there is a difference between Click Through Rate (CTR) between Ads one and two.
t-test for two populations:
Similar to the first question, we know the sample mean of each experiment is the CTR we need to compare. According to CTR, the sample mean is limiting normal distribution. Then we have
If the click-through rate is not constant and is in fact randomly distributed (e.g. normal distributed), how does this affect the standard deviation of the sample mean CTR and consequently the validity of the test result.
My concern is that if use the above distribution and the actual standard deviation is higher we will end up accepting the alternative hypothesis incorrectly. Anecdotally, when we have run A/A tests (i.e. the variants are the same), we appear to often accept the alternative hypothesis more often than you would expect to and I want to find out whether this could be due to the assumptions around the click-through rate being constant.