I have a data in which there is a high degree of variability.
My Objective is to do an AB test to check the behavior change due to new changes.
- All samples has shown historically high and low performances. This means if I take any 2 cohorts randomly, they show vast historical comparison difference Following is the example for weekly comparison. Same behavior holds true for monthly and daily too.
W10: 1.92% ... .. .
No matter I tried to segment users by their aggregate behavior and see the difference, its still this level of variability is observed.
I'm looking for clearly difference to be consistent no matter in which ever direction. Either its positive or negative. And, with small range of magnitude.
Can someone please suggest the alternative methods to minimize this difference?