1
$\begingroup$

enter image description here

I’m a developer reading the book Bayesian statistics the fun way

In chapter 15 they use hypothesis testing using Monte Carlo simulation to pick random values from two intercepting beta distributions

I want to know if they could have used PDF/CDF instead? Why did they choose Monte Carlo when you can get a solid value from CDf/PDF method of getting the probability from a distribution? And compare which one is better in the case needed?

We’ve covered the basics of parameter estimation pretty well at this point. We’ve seen how to use the PDF, CDF, and quantile functions to learn the likelihood of certain values, and we’ve seen how to add a Bayesian prior to our estimate. Now we want to use our estimates to compare two unknown parameters. The accurate answer to which email variant generates a higher click-through rate lies somewhere in the intersection of the distributions of A and B. Fortunately, we have a way to figure it out: a Monte Carlo simulation Clearly, our data suggests that variant B is superior, in that it garners a higher conversion rate. However, from our earlier discussion on parameter estimation, we know that the true conversion rate is one of a range of possible values. We can also see here that there’s an overlap between the possible true conversion rates for A and B. What if we were just unlucky in our A responses, and A’s true conversion rate is in fact much higher? What if we were also just lucky with B, and its conversion rate is in fact much lower? It’s easy to see a possible world in which A is actually the better variant, even though it did worse on our test. So the real question is: how sure can we be that B is the better variant? This is where the Monte Carlo simulation comes in.

$\endgroup$
1
  • $\begingroup$ I understand that what is shown in the plot are the distributions of the conversion rates of a Beta-binomial model, that is, if the process of a user converting or not is modelled with a Bernoulli distribution, those would be the Beta distributions of the parameter of the Bernoulli distribution of the A and B approaches. Can you describe in detail how you would apply the pdf/cdf in this case? $\endgroup$
    – noe
    Dec 21, 2023 at 17:55

1 Answer 1

1
$\begingroup$

You are interested in the distribution of $D := \theta_{A} - \theta_{B} | W$, where $\theta_{A}$ and $\theta_{B}$ are parameters from your model representing the posterior click-through rates under variants A and B, respectively. $W$ is meant to represent "the data", i.e. we are conditioning on all of our available information and are working with the posterior.

To avoid Monte Carlo simulation, you have to derive the probability distribution of $D$, which means you have to work with the joint posterior density $f_{\theta_{A}, \theta_{B} \ | {W}} \ (a, b)$. The problem is that we don't often know what the joint density is. Deriving it in some cases is possible for specific distributions and under assumptions like independence between $\theta_{A}$ and $\theta_B$, but in general, it's often easier to just use Monte Carlo simulation since in practice, you will have:

  1. realized samples from the joint posterior distribution via. MCMC, of which you can estimate any quantity you wish, OR
  2. an approximation to the true joint posterior distribution, of which you can simulate from/use that approximation directly if closed-form expressions exist for it

Also: without sounding too pedantic, note that the idea of a "hypothesis test" is very much a frequentist and not a Bayesian way of thinking.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.