7
$\begingroup$

Apologies if this is very broad question, what I would like to know is how effective is A/B testing (or other methods) of effectively measuring the effects of a design decision.

For instance we can analyse user interactions or click results, purchase/ browse decisions and then modify/tailor the results presented to the user.

We could then test the effectiveness of this design change by subjecting 10% of users to the alternative model randomly but then how objective is this?

How do we avoid influencing the user by the model change, for instance we could decided that search queries for 'David Beckham' are probably about football so search results become biased towards this but we could equally say that his lifestyle is just as relevant but this never makes it into the top 10 results that are returned.

I am curious how this is dealt with and how to measure this effectively.

My thoughts are that you could be in danger of pushing a model that you think is correct and the user obliges and this becomes a self-fulfilling prophecy.

I've read an article on this: http://techcrunch.com/2014/06/29/ethics-in-a-data-driven-world/ and also the book: http://shop.oreilly.com/product/0636920028529.do which discussed this so it piqued my interest.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

In A/B testing, bias is handled very well by ensuring visitors are randomly assigned to either version A or version B of the site. This creates independent samples drawn from the same population. Because the groups are independent and, on average, only differ in the version of the site seen, the test measures the effect of the design decision.

Slight aside: Now you might argue that the A group or B group may differ in some demographic. That commonly happens by random chance. To a certain degree this can be taken care of by covariate adjusted randomization. It can also be taken care of by adding covariates to the model that tests the effect of the design decision. It should be noted that there is still some discussion about the proper way to do this within the statistics community. Essentially A/B testing is an application of a Randomized Control Trial to website design. Some people disagree with adding covariates to the test. Others, such as Frank Harrel (see Regression Modeling Strategies) argue for the use of covariates in such models.

I would offer the following suggestions:

  • Design the study in advance so as to take care of as much sources of bias and variation as possible.
  • Let the data speak for itself. As you get more data (like about searches for David Beckham), let it dominate your assumptions about how the data should be (as how the posterior dominates the prior in Bayesian analysis when the sample size becomes large).
  • Make sure your data matches the assumptions of the model.
Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the answer, it also makes me think about the ethics of when you do an experiment in order to see its influence such as the recent fabeook issue, perhaps that itself would make a good question as to the moral implications. $\endgroup$
    – EdChum
    Jul 23, 2014 at 13:55
  • $\begingroup$ You're welcome. Ethics is something we deal with a lot in Biostatistics due to the history of medical research. As a statistician/data scientist, I would argue that it is ethical to give an accurate portrait of the data and to not torture it into confessing. A good place to start for the ethics of trials, from the medical point of view is the Nurenburg Code. It certainly has application for what Facebook did. $\endgroup$
    – Christopher Louden
    Jul 23, 2014 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.