I am working on a planned sequence of n independent A/B tests(=run a maximum of n tests or stop earlier if a good improvement is found) and in order to keep the significance level within an acceptable level(=0.05) I'm considering ways of controlling the false positives rate while keeping the sample size as low as I can(it's for a low traffic website)
I'm aware of the Bonferroni, Benjiamimi-Hochberg and related methods designed to control the proportion of false positives in the multiple comparisons situation. However the problem of computing the sample size still remains, the simplest approach seems the one of computing the required sample size starting from alpha = 0.05/n as dictated by the Bonferroni correction and then perhaps use Benjiamimi-Hochberg for the actual testing. Is there a better way of computing the sample size when Benjiamimi-Hochberg is used?
Further as the A/B tests are designed to test possible improvements I was thinking that if the sequential tests are based on independent samples then then probability of two consecutive false positives would be alpha^2 = 0.025. My understanding is that the sequential tests would be independent if the respective samples do not refer to the same users. These could for example be users that joined the website after the previous test.
If the above thinking makes some sense, I could use a "weak" Bonferroni (i.e. alpha' = alpha/(n*0.2) )correction in order to run tests with a reasonable power and lower sample size and once a positive is found(the null hypothesis is rejected), I could repeat the test and if it's positive again then the result is positive with p < 0.025 which means that I would accept the result. This approach would allow for some control over the sample size but... is it sound?
Any opinion would be greatly appreciated. Thanks in advance. AMAR