6
$\begingroup$

I asked a data science question regarding how to decide on the best variation of a split test on the Statistics section of StackExchange. I hope I will have better luck here. The question is basically, "Why is mean revenue per user the best metric to make your decision on in a split test?"

The original question is here: https://stats.stackexchange.com/questions/107599/better-estimator-of-expected-sum-than-mean

Since it was not well received/understood I simplified the problem to a discrete set of purchases and phrased it as a classical probability problem. That question is here: https://stats.stackexchange.com/questions/107848/drawing-numbered-balls-from-an-urn

The mean may be the best metric for such a decision but I am not convinced. We often have a lot of prior information so a Bayesian method would likely improve our estimates. I realize that this is a difficult question but Data Scientists are doing such split tests everyday.

$\endgroup$
  • $\begingroup$ Most of your question here is describing a problem with your question. The text does not contain your question. Please in-line it and improve the text if needed. I don't think you need to summarize the history on the other site. $\endgroup$ – Sean Owen Aug 8 '14 at 12:20
6
$\begingroup$

If you've got prior information then you should certainly not use simple mean in a split test. I assume you're trying to just predict which group will produce the greatest amount of revenue overall, by trying to emulate the underlying distribution.

Firstly, it's worth noting that any metrics you choose will actually reduce to mean in a pretty trivial way. Eventually mean will necessarily work out, though using a standard bayesian method to estimate the mean is probably your best bet.

If you've got a prior then using a standard bayesian approach to update the prior on your mean revenue is probably the best way to do it. Basically, just take the individual results you get and update a multinomial distribution representing your prior in each case.

If you want some more full background on multinomial distributions as bayesian priors are pretty well, this Microsoft paper does a pretty good job of outlining it. In general, I wouldn't care so much about the fact that your distribution is technically discrete, as a multinomial distribution will effectively interpolate across your solution space, giving you a continuous distribution that is a very good approximation of your discrete space.

$\endgroup$
  • $\begingroup$ Yes this is what I was looking for. I would be interested to see if this reduces to something like a Welch t-test because it may illuminate some complications. Eq. 52 in the paper you gave is nearly exactly what I want except it is not just a multinomial but a multinomial called a number of times for each user. I think the problem boils down to a generalized Behrens–Fisher problem. I am however a little out of my depth. Do you think you could sketch out a solution for this particular case? I know there are others on Cross Validated interested in the solution. $\endgroup$ – Keith Jul 28 '14 at 11:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.