We noticed we had a biased sample in our A/B test and was wondering if difference-in-differences would help us make valid conclusions about the data, or if there was another way to proceed.

We ran an new experiment on our site, where we offered 50% of our users a new feature. We assigned users with odd ids into the experiment group and users with even ids into the control group and then ran the experiment. However, we saw that even prior to running the experiment, there was a statistically difference between the two groups. We think this is because we ran many experiments where we segmented based on odd/even of the id, so people in the experiment group have seen many treatments.

So lesson learned, we'll flip a coin next time instead of using the id. However, we'd like to see if we can still make inferences from the current experiment. I've heard of something called difference-in-differences. Would this work in this case, or is there a different approach that would work better? Ideally, we don't want to scrap the test and start over since 50% of our users have already seen the new feature.


Why are you hesitant to start over? A/B testing is double-edged sword because it is a relatively straightforward exercise (as these things go) but it has the power to completely pivot your business in a very short amount of time. Plus keep in mind that the thresholds you are seeking to surpass can be "tempermental" (for the lack of a better word) where a z-value of (for example) 1.92 can't be, "oh, that's close enough . . . ".

As a data scientist, I would never want to sign my name to an experiment that I didn't have complete confidence in both the design and execution. If there is anything that can affect your final scores, you should absolutely consider scrapping it and starting over.


At this point in the experiment, nothing you can do can make the two groups equal. What you can do is make assumptions about how the two groups will behave.

If you can safely assume that previous treatments that you have applied using the even/odd sampling method will not interact with the new A/B test, then a difference-in-differences method would be appropriate.

As a simple example of when this assumption (and a difference-in-differences) is inappropriate: Let's say in your first test you send group A (the odds) a book titled "The importance of eating healthy." In the second test, you send group A (the odds) dietary supplements. Group A has been compromised by prior treatments that could impact the effects on the second test.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.