How to control false positives in sequential A/B testing while keeping a low sample size?

Question

I am working on a planned sequence of n independent A/B tests(=run a maximum of n tests or stop earlier if a good improvement is found) and in order to keep the significance level within an acceptable level(=0.05) I'm considering ways of controlling the false positives rate while keeping the sample size as low as I can(it's for a low traffic website)

I'm aware of the Bonferroni, Benjiamimi-Hochberg and related methods designed to control the proportion of false positives in the multiple comparisons situation. However the problem of computing the sample size still remains, the simplest approach seems the one of computing the required sample size starting from alpha = 0.05/n as dictated by the Bonferroni correction and then perhaps use Benjiamimi-Hochberg for the actual testing. Is there a better way of computing the sample size when Benjiamimi-Hochberg is used?

Further as the A/B tests are designed to test possible improvements I was thinking that if the sequential tests are based on independent samples then then probability of two consecutive false positives would be alpha^2 = 0.025. My understanding is that the sequential tests would be independent if the respective samples do not refer to the same users. These could for example be users that joined the website after the previous test.

If the above thinking makes some sense, I could use a "weak" Bonferroni (i.e. alpha' = alpha/(n*0.2) )correction in order to run tests with a reasonable power and lower sample size and once a positive is found(the null hypothesis is rejected), I could repeat the test and if it's positive again then the result is positive with p < 0.025 which means that I would accept the result. This approach would allow for some control over the sample size but... is it sound?

Any opinion would be greatly appreciated. Thanks in advance. AMAR

Answer 1

=run a maximum of n tests or stop earlier if a good improvement is found

Don't do that. Don't stop before n tests have been run if you like what you've seen so far....

http://www.evanmiller.org/how-not-to-run-an-ab-test.html

In general, however, figuring out how big an N you need given an expected effect size and alpha level is the domain of power analysis. Read more here:

http://www.uta.edu/faculty/sawasthi/Statistics/stpowan.html

Answer 2

Like Emre said, if you have a certain type I and type II error rates that need to be met, you can consider the Sequential Probability Ratio Test or other methods like Distilled Sensing.

If you were conducting the A/B test in an online fashion, i.e. decisions need to be made for each location as they arrive, then you can consider some online FDR procedures (generalized alpha-investing methods) like LORD, LOND, $\alpha$-investing or spending and GAI++.

Bonferroni type corrections are for FWER control and tend to be overly conservative.