What is the meaning of likelihood?

Question

I am studying Bayes probability applied to machine learning, and I have encoutered the concept of likelihood, which I don't understand.

I have seen that the Bayes rule is:

$P(A|B)=\frac{P(B|A)P(A)}{P(B)}$

where $P(B|A)P(A)$ is the conditional probability, while $P(B|A)$ is the likelihood, but I don't understand what the likelihood is.

Can somebody please help me?

Answer 1

Theory

Let us use notations that are less confusing than in your question: the likelihood of a probability distribution $P$ indexed by hyperparameters $\theta$ (namely, $P_\theta$), given a set of observations ${x_i}$, is the probability that the observations were yielded by this distribution, given $\theta$. This translates in: $$\mathcal{L}(\theta \mid x) = P_\theta(X_1=x_1, \ldots, X_n = x_n)$$ which can also be written: $P(X = x\mid\theta)$

The link with Bayes statistics is the reversal of conditions: $\theta\mid x$ into $x\mid\theta$.

The highest the probability, the more likely it is that the observations were yielded by this distribution.

Example

Consider a set of 10 heads/tails draws, which result in 7 heads and 3 tails. You can define the likelihood that the coin is balanced ($p_\text{heads} = p_\text{tails} = 0.5$), given the results, by: $$\mathcal{L}_\text{balanced} = P(\text{7 heads, 3 tails} \mid p_\text{heads} = 0.5) = \binom{10}{7}\left(\frac{1}{2}\right)^7\left(\frac{1}{2}\right)^3 \sim 0.117$$

And the likelihood that the probability of heads is 0.8 is: $$\mathcal{L}_\text{balanced} = P(\text{7 heads, 3 tails} \mid p_\text{heads} = 0.8) = \binom{10}{7}0.8^70.2^3 \sim 0.203$$

From this, you can tell that it is more likely that the coin has 80% probability of heads, than it is balanced.

Answer 2

All of the answers here, including the accepted one, are conspicuously confused. I down-voted the accepted answer but downvotes of users who lack reputation in this "community" are not counted. I have a reputation of more than 200,000 (two-hundred-thousand) on math.stackexchange.com and I have a Ph.D. in statistics, but none of that counts here.

The question says $P(B\mid A)$ is the likelihood, but it ought to say instead that $A\mapsto P(B\mid A)$ is the likelihood. I.e. it's this probability as a function of $A,$ not of $B.$

It also says $P(B\mid A)P(A)$ is the conditional probability. That is not right. $P(B\mid A)$ is a conditional probability. $P(B\mid A)P(A)$ is $P(A\ \&\ B).$

Now suppose you have $$ \Pr(A_1) = \frac 1 {20}, \quad \Pr(A_2) = \frac 2 {20}, \quad \Pr(A_3) = \frac{17}{20} $$ and $A_1,A_2,A_3$ are mutually exclusive. This is a prior probability distribution.

Further suppose that $$ \Pr(B\mid A_1) = \frac 9 {10}, \quad \Pr(B\mid A_2)= \frac 2 3, \quad \Pr(B\mid A_3) = \frac 1 2. $$ Note well:

• These three probabilities do not add up to $1.$ What is expressed here is not a probability distribution.

• It is a function of $\text{“}A\text{''},$ a variable whose value may be either $A_1,$ $A_2,$ or $A_3.$

• The likelihood is $\Pr(B\mid A)$ as a function of $A,$ not of $B.$

Bayes's theorem says: When one multiplies (pointwise) the prior probability distribution by the the likelihood and then normalizes, one gets the posterior probability distribution, i.e. the distribution conditional on the data. The "data" is the observed event $B.$

Thus \begin{align} & \left( \tfrac 1 {20}, \tfrac 2 {20}, \tfrac{17}{20} \right) \times\left( \tfrac 9{10}, \tfrac 2 3, \tfrac 1 2 \right) \\[8pt] = {} & \left( \tfrac 9 {200}, \tfrac 2 {60}, \tfrac{17}{40} \right) \\[8pt] \propto {} & \left( 27, 20, 255 \right) \\ & \text{(Here I multiplied all three components} \\ & \phantom{(} \text{by 600, which is the l.c.m. of the denominators.)} \\[8pt] \propto {} & \left( \tfrac{27}{302}, \tfrac{20}{302}, \tfrac{255}{302} \right) \\ & \text{(Here I normalized, i.e. divided by} \\ & \phantom{(} 27+20+255 = 302 \text{ so that the} \\ & \phantom{(} \text{sum of the three components is $1.$)} \\[8pt] = {} & \left( \Pr(A_1\mid B), \Pr(A_2\mid B), \Pr(A_3\mid B) \right) \\[8pt] = {} & \textbf{the posterior probability distribution.} \end{align}

Often one sees something like $$ X\sim\operatorname{Binomial}(4,p), $$ so that, for example $$ \Pr(X=2) = \binom 4 2 p^2 (1-p)^{4-2}. $$ Then if the data consists of the observation that $X=2,$ then the likelihood function is $$ L(p) = \binom 4 2 p^2(1-p)^{4-2}. $$ This is a function of $p,$ not of a variable whose values can be the possible values of $X,$ which in this case are $0,1,2,3,4.$

Answer 3

Likelihood and probability are two very different concepts:

One talk about probabilities when the distribution is already known and one want to know how probable an event is.

Likelihood on the other hand is usually much more experimental. It is used when, given some results, one want to know how likely it is that those results fit a specific distribution.

In other words, probability has to do with uncertainty on events while likelihood has to do with uncertainty on distributions.

Here is a great video from stat quest explaining the difference between the two concepts: https://youtu.be/pYxNSUDSFH4