0
$\begingroup$

I am reading a paper regarding A/B test continuous monitoring. https://alexdeng.github.io/public/files/continuousMonitoring.pdf

In the abstract part, it states

A crucial problem of Null Hypothesis Statistical Testing (NHST), the backbone of A/B testing methodology, is that experimenters are not allowed to continuously monitor the results and make decisions in real time.

But without explain why this is the case.

So my question is: Why can't we run NHST based A/B testing continuously and make real time decision based on the result?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Because that would lead immediately to data dredging. Imagine you have a sample and suppose you perform the test on the entire sample and get low effects, insignificant p-value and so on. If you would follow an iterative procedure in which you start with a single observation from the sample and while repeatedly add one occurrence you also perform a test, there are high chances to get a significant result at some point in time. Simply due to the randomness of the order of occurrences in the sample. The problem would be that you would do repeatedly a test with no correction for your criteria.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.