5
$\begingroup$

After 6 months of AB testing on our CRM tool (Oracle Responsys, but this could be true with anyone), the test exhibited some weird results so we decided to pause everything, and to make some good old AA testing.

AA testing consists in dividing randomly the users between two branches, making both branches have the exact same experience, and testing that conversion rates on both branches are not significantly different, which would mean that both branches are not actually treated the same (or that the population is not uniformly distribued between two branches)

As you can imagine, this did not happen at all. We sent roughly 3500 mails to each branch, and one of these exhibits a 40% conversion while the other only has 32%, which is a difference that has a p-value of the order of 10^-12

Furthermore, we partitionned the time in 4 periods, each one of these exhibit a very significant difference with the other one.

Results of AA testing

Now what am I supposed to do? I would like to have a theoretical discussion about this with Oracle's support, but of course, everybody that sees an AA testing go wrong will believe you did not respect a proper methodology.

My question : I would like some pointers of people with more experience than me in data science to help me to :

- Express my problem in the clearest possible way to get the company that sells me this solution to take an interest in the issue here

- Understand if I am forgetting to check anything on my side to explain this discrepancy

The exact process that is used from our server to the final client is :

  • we send every x hours a list of a few hundred people to Responsys

  • Responsys is supposed to affect them in an independent and identically distributed way using random numbers generated with java.util.random.nextfloat()

  • then the list is filtered to remove undeliverable people, and each branch is sent the exact same email

Oracle is pretty laconic in his description, but I think the following lines from their documentation is supposed to describe an i.i.d. uniform distribution.

During launch, for each email recipient:

a. generate a random number ranging in [0, 1] using java.util.Random.nextFloat()

b. decide the bucket this random number is in

c. send the campaign in that bucket to the recipient

Improve this question
$\endgroup$
3
  • $\begingroup$ I ran it once, during 13 days. $\endgroup$
    – WNG
    Aug 9, 2016 at 15:28
  • $\begingroup$ "Get the company that sells me this solution to take an interest in the issue here" - That's out of scope here. $\endgroup$
    – D.W.
    Mar 8, 2017 at 21:26
  • $\begingroup$ I meant "Which angle could I take to present it to the support so that they don't dismiss my tickets as the ramblings of a non-statistician". I'll modify my wording to make it clearer that I don't expect to use stackexchange as a soapbox $\endgroup$
    – WNG
    Mar 8, 2017 at 22:15

2 Answers 2

Reset to default
4
$\begingroup$

I 'solved' the issue with Oracle's support, I still post an answer here because some of what I encountered can be useful : According to Oracle's support, this bug in AB testing programs was known for the last 9 month, but corrected only this week-end.

So my problem is supposedly solved (I will re-launch a test and see if results are now consistent), but I still have some pointers if someone encounters the same problem I did :

  • Make sure this is not a statistical artifact : test your hypothesis on several independant periods. I used a Fischer test because it is exact for a binomial framework, it is what you should use if your population/conversion rate is too small for a normal approximation

  • Respect hypothesis testing framework : start your test after you formulated your hypothesis

  • Get a very very low p-value : No one will listen to your complaint about faulty AB testing if you come up with a 3% p-value, and they would be right. A bayesian analysis probably points to a false positive.

  • Make sure the random numbers that are used to split the populations are actually random (and not, say a 50/50 split)

  • Imagine all the possibilities that your reporting might be wrong. I thought I had imagined them all, but if you read below the real reason behind the bug, you will find that faulty reporting can be hidden very far away in parts of the environment you don't control or even have a view in.

I have the actual answer from Oracle that describes the bug that happened, in case someone has the exact same problem or wants to know how this kind of bug happens :

For programs triggered by event, their engine does not have the capability to associate the different open and clicks events to the correct email. In the case several emails have been sent through the same program, Responsys assumes the event concerns the earliest email that has been sent through this program.

In my case, the program could have been sent several times during the period.

In the period where I was not splitting my emails through two branches, the name of the campaign I used was already A (the same than one of my branches in my testing period)

You can see where this is going : Every customer opening any email was marked as having read the "A" email, as long as they had been into the program in the 3 months before I started my AB testing, which caused a faulty reporting.

So actually both branches were sent to the correct customers, and it is just a reporting problem that comes from an approximative algorithm on Responsys side.

The workaround I will try now is simply to create two brand new "campaign" objects before starting a test rather than using a campaign that already existed.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ It would be interesting to hear what the bug was. I imagine that they likely were using the same seed value each time and the random number generator was thus not really random. $\endgroup$
    – AN6U5
    Aug 10, 2016 at 7:41
  • $\begingroup$ I would guess some failure to treat split evenly, either pre-sending or post sending (e.g. accidentally assign some responses to A when in fact they were in B). If this also applied to regular AB tests, it would be a very serious bug indeed, making the use of the whole product questionable. $\endgroup$
    – Neil Slater
    Aug 10, 2016 at 13:13
  • 1
    $\begingroup$ I added the bug explanation, which totally makes sense with respect to what happened to me. I am running a third AA testing that hopefully will at last give me the dull results I expect $\endgroup$
    – WNG
    Aug 10, 2016 at 21:56
2
$\begingroup$

Using frequentist hypothesis testing in this manner (using the p-value to determine pass/fail) has serious drawbacks even in bug-free software. My suggestion is to use some form of Bayesian A-B testing. There is much nerd debate about this (frequentist vs. Bayesian A/B testing).

First, some references. Then we'll have a look at your AA test with the online calculator.

On using p-values as pass/fail criteria:

On Bayesian A/B testing:

enter image description here

enter image description here

In your case the frequentist and Bayesian approaches agree-- your "A" is not equal to your other "A". It appears that there are one or more currently unseen variable(s) you need to take into account. If it was a question of insufficient data, the two peaks would not be so nicely separated. I'd be interested in hearing other possible explanations.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.