2
$\begingroup$

I have a data in which there is a high degree of variability.

My Objective is to do an AB test to check the behavior change due to new changes.

  1. All samples has shown historically high and low performances. This means if I take any 2 cohorts randomly, they show vast historical comparison difference Following is the example for weekly comparison. Same behavior holds true for monthly and daily too.

W1: -10.04%

W2: 3.9%

W3: -4.2%

W4: -3.7%

W5: 5.4%

W6: -2.6%

W8: 0.8%

W9: -2.3%

W10: 1.92% ... .. .

  1. No matter I tried to segment users by their aggregate behavior and see the difference, its still this level of variability is observed.

  2. I'm looking for clearly difference to be consistent no matter in which ever direction. Either its positive or negative. And, with small range of magnitude.

Can someone please suggest the alternative methods to minimize this difference?

Improve this question
$\endgroup$
1
  • $\begingroup$ I want to consider moving average to be one of the possibilities to smooth-en the curve. $\endgroup$
    – MogambO
    Nov 15, 2019 at 4:32

1 Answer 1

Reset to default
0
$\begingroup$

Your ability to detect a difference depends on the relative standard deviation of your data. Sometimes the noise is due to seasonal effects, which an AB test does an excellent job in correcting. However if the metric itself is intrinsically high variance, you will need a large sample size, or adjust your significance and power parameters.

Run a check of your standard deviation and do a power calculation to evaluate your situation.

Keep in mind that there are some statistical distributions which have variances approaching infinite.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.