After 6 months of AB testing on our CRM tool (Oracle Responsys, but this could be true with anyone), the test exhibited some weird results so we decided to pause everything, and to make some good old AA testing.
AA testing consists in dividing randomly the users between two branches, making both branches have the exact same experience, and testing that conversion rates on both branches are not significantly different, which would mean that both branches are not actually treated the same (or that the population is not uniformly distribued between two branches)
As you can imagine, this did not happen at all. We sent roughly 3500 mails to each branch, and one of these exhibits a 40% conversion while the other only has 32%, which is a difference that has a p-value of the order of 10^-12
Furthermore, we partitionned the time in 4 periods, each one of these exhibit a very significant difference with the other one.
Now what am I supposed to do? I would like to have a theoretical discussion about this with Oracle's support, but of course, everybody that sees an AA testing go wrong will believe you did not respect a proper methodology.
My question : I would like some pointers of people with more experience than me in data science to help me to :
- Express my problem in the clearest possible way to get the company that sells me this solution to take an interest in the issue here
- Understand if I am forgetting to check anything on my side to explain this discrepancy
The exact process that is used from our server to the final client is :
we send every x hours a list of a few hundred people to Responsys
Responsys is supposed to affect them in an independent and identically distributed way using random numbers generated with java.util.random.nextfloat()
then the list is filtered to remove undeliverable people, and each branch is sent the exact same email
Oracle is pretty laconic in his description, but I think the following lines from their documentation is supposed to describe an i.i.d. uniform distribution.
During launch, for each email recipient:
a. generate a random number ranging in [0, 1] using java.util.Random.nextFloat()
b. decide the bucket this random number is in
c. send the campaign in that bucket to the recipient