3
$\begingroup$

I am studying Bayes probability applied to machine learning, and I have encoutered the concept of likelihood, which I don't understand.

I have seen that the Bayes rule is:

$P(A|B)=\frac{P(B|A)P(A)}{P(B)}$

where $P(B|A)P(A)$ is the conditional probability, while $P(B|A)$ is the likelihood, but I don't understand what the likelihood is.

Can somebody please help me?

$\endgroup$
-1
$\begingroup$

Theory

Let us use notations that are less confusing than in your question: the likelihood of a probability distribution $P$ indexed by hyperparameters $\theta$ (namely, $P_\theta$), given a set of observations ${x_i}$, is the probability that the observations were yielded by this distribution, given $\theta$. This translates in: $$\mathcal{L}(\theta \mid x) = P_\theta(X_1=x_1, \ldots, X_n = x_n)$$ which can also be written: $P(X = x\mid\theta)$

The link with Bayes statistics is the reversal of conditions: $\theta\mid x$ into $x\mid\theta$.

The highest the probability, the more likely it is that the observations were yielded by this distribution.

Example

Consider a set of 10 heads/tails draws, which result in 7 heads and 3 tails. You can define the likelihood that the coin is balanced ($p_\text{heads} = p_\text{tails} = 0.5$), given the results, by: $$\mathcal{L}_\text{balanced} = P(\text{7 heads, 3 tails} \mid p_\text{heads} = 0.5) = \binom{10}{7}\left(\frac{1}{2}\right)^7\left(\frac{1}{2}\right)^3 \sim 0.117$$

And the likelihood that the probability of heads is 0.8 is: $$\mathcal{L}_\text{balanced} = P(\text{7 heads, 3 tails} \mid p_\text{heads} = 0.8) = \binom{10}{7}0.8^70.2^3 \sim 0.203$$

From this, you can tell that it is more likely that the coin has 80% probability of heads, than it is balanced.

$\endgroup$
5
  • 1
    $\begingroup$ I downvoted this based on its highly confused first sentence. $\endgroup$ – Michael Hardy Dec 26 '19 at 19:08
  • $\begingroup$ @MichaelHardy Is it wrong though? Anyway the answer is sufficient in itself, so I'll delete the first sentence if you feel it's confusing. $\endgroup$ – Romain Reboulleau Dec 26 '19 at 23:36
  • $\begingroup$ You wrote "the likelihood of a probability distribution". That is grossly wrong. $\endgroup$ – Michael Hardy Dec 27 '19 at 7:16
  • $\begingroup$ @MichaelHardy OK. Don't be mistaken: I'm really trying to learn here, but you're telling me what's wrong, not what's right :) Could you maybe suggest edits to my post in order to correct that kind of mistake? $\endgroup$ – Romain Reboulleau Dec 27 '19 at 8:30
  • $\begingroup$ Look at the example of a likelihood function in my posted answer. You can see that that is not a probability distribution. And when you get into continuous distributions or even parametrized families of discrete distributions with a continuous parameter, the difference gets deeper. $\endgroup$ – Michael Hardy Dec 27 '19 at 20:02
2
$\begingroup$

All of the answers here, including the accepted one, are conspicuously confused. I down-voted the accepted answer but downvotes of users who lack reputation in this "community" are not counted. I have a reputation of more than 200,000 (two-hundred-thousand) on math.stackexchange.com and I have a Ph.D. in statistics, but none of that counts here.

The question says $P(B\mid A)$ is the likelihood, but it ought to say instead that $A\mapsto P(B\mid A)$ is the likelihood. I.e. it's this probability as a function of $A,$ not of $B.$

It also says $P(B\mid A)P(A)$ is the conditional probability. That is not right. $P(B\mid A)$ is a conditional probability. $P(B\mid A)P(A)$ is $P(A\ \&\ B).$

Now suppose you have $$ \Pr(A_1) = \frac 1 {20}, \quad \Pr(A_2) = \frac 2 {20}, \quad \Pr(A_3) = \frac{17}{20} $$ and $A_1,A_2,A_3$ are mutually exclusive. This is a prior probability distribution.

Further suppose that $$ \Pr(B\mid A_1) = \frac 9 {10}, \quad \Pr(B\mid A_2)= \frac 2 3, \quad \Pr(B\mid A_3) = \frac 1 2. $$ Note well:

  • These three probabilities do not add up to $1.$ What is expressed here is not a probability distribution.

  • It is a function of $\text{“}A\text{''},$ a variable whose value may be either $A_1,$ $A_2,$ or $A_3.$

  • The likelihood is $\Pr(B\mid A)$ as a function of $A,$ not of $B.$

Bayes's theorem says: When one multiplies (pointwise) the prior probability distribution by the the likelihood and then normalizes, one gets the posterior probability distribution, i.e. the distribution conditional on the data. The "data" is the observed event $B.$

Thus \begin{align} & \left( \tfrac 1 {20}, \tfrac 2 {20}, \tfrac{17}{20} \right) \times\left( \tfrac 9{10}, \tfrac 2 3, \tfrac 1 2 \right) \\[8pt] = {} & \left( \tfrac 9 {200}, \tfrac 2 {60}, \tfrac{17}{40} \right) \\[8pt] \propto {} & \left( 27, 20, 255 \right) \\ & \text{(Here I multiplied all three components} \\ & \phantom{(} \text{by 600, which is the l.c.m. of the denominators.)} \\[8pt] \propto {} & \left( \tfrac{27}{302}, \tfrac{20}{302}, \tfrac{255}{302} \right) \\ & \text{(Here I normalized, i.e. divided by} \\ & \phantom{(} 27+20+255 = 302 \text{ so that the} \\ & \phantom{(} \text{sum of the three components is $1.$)} \\[8pt] = {} & \left( \Pr(A_1\mid B), \Pr(A_2\mid B), \Pr(A_3\mid B) \right) \\[8pt] = {} & \textbf{the posterior probability distribution.} \end{align}

Often one sees something like $$ X\sim\operatorname{Binomial}(4,p), $$ so that, for example $$ \Pr(X=2) = \binom 4 2 p^2 (1-p)^{4-2}. $$ Then if the data consists of the observation that $X=2,$ then the likelihood function is $$ L(p) = \binom 4 2 p^2(1-p)^{4-2}. $$ This is a function of $p,$ not of a variable whose values can be the possible values of $X,$ which in this case are $0,1,2,3,4.$

$\endgroup$
1
  • 1
    $\begingroup$ Hi, I don't think putting quotes around "community" is a good way to start in our "community". We would surely welcome improvements of existing answers, or comments on why they are confusing. $\endgroup$ – Romain Reboulleau Dec 26 '19 at 23:45
1
$\begingroup$

Likelihood and probability are two very different concepts:

One talk about probabilities when the distribution is already known and one want to know how probable an event is.

Likelihood on the other hand is usually much more experimental. It is used when, given some results, one want to know how likely it is that those results fit a specific distribution.

In other words, probability has to do with uncertainty on events while likelihood has to do with uncertainty on distributions.

Here is a great video from stat quest explaining the difference between the two concepts: https://youtu.be/pYxNSUDSFH4

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.