I am confused by the definition of the likelihood function in different contexts.

In the case of linear and logistic regression, it is defined as y given x

In the case naive bayes and LDA, it is defined over x and y

In the case of the EM algorithm (with observed variable x and unobserved variable z), it is defined over x

How do we know over what it should be defined?


1 Answer 1


I also found confusing the definition of likelihood the first time I encountered it. The only thing you need to remember is that likelihood refers to the probability of your data given a parameter. That is,

$$L(\theta| X) = P(X|\theta)$$

As you can see, the likelihood focuses on the parameter, so you're asking "What is the likelihood of this parameter given some observations?" In this case, the phrase "given some observation" assumes the observations $X$ are fixed and the parameter $\theta$ is not fixed. On the other hand, if you describe the same in terms of probability, you're trying to measure the probability of some observations $X$ given a fixed parameter $\theta$. It is only a matter of usage.

In the case of logistic regression, the likelihood function is described as:

$$L(\beta| y) = \prod_{i}^{N} \pi_{i}^{y_{i}}(1-\pi_{i})^{n_{i} - y_{i}}$$

where $\beta$ is the vector of parameters $\{\pi_{i}| i=1,...,N \}$and $y$ is the vector of Bernoulli variables. As you can see, this is the same (by definition) as $P(y|\beta)$ and that is what you have in the equation above: The product of the probabilities for each observation $y_{i}$ given its parameter $\pi_{i}$. As usual, each probability $\pi_{i}$ is equal to the inverse of the logit function.

In Naive Bayes, you have the same basic idea. The likelihood is defined as $L(p_{k}|x) = \prod_{i}^{N}p_{ki}^{x_{i}}(1-p_{ki})^{(1-x_{i})}$ where $p_{k}$ is a vector of parameters describing the probabilities of a class $k$ generating a word $x_{i}$).

In the EM algorithm, the likelihood is again $L(\theta|X,Z)$, where $\theta$ are your parameters, $X$ your observations and $Z$ are latent variables. The fact that you have a latent variables shouldn't confuse you, because this variable will be "integrated out" in the expectation step of the algorithm.

In the previous equations, the calculation you need to do is self-evident if you think about it starting from your observations: First, I have some observations. Then I want to calculate the probability of obtaining these observation using some model. Most models require parameters to be fitted. So I want to calculate the probability of these observations given those parameters. That is, $p(X|\theta)$. Now, if you need to talk about likeliihood, then you simply write $L(\theta|X)$.

  • $\begingroup$ thank you very much for your answer. In the case of Naive Bayes, don't we have the likelihood p(x,y) over both the examples x and the class labels y? (as in cs229.stanford.edu/notes/cs229-notes2.pdf), whereas in logistic regression it is only over y: p(y|x) (cs229.stanford.edu/notes/cs229-notes1.pdf) $\endgroup$ Feb 15, 2015 at 13:17
  • $\begingroup$ In the case of Naive Bayes, that is the joint probability $p(x,y)$ and they use in page 10 what they call the "joint likelihood". In the logistic regression example, the notation in page 18 might be confusing because they have $p(y|x;\theta)$. For a moment, forget about $x$ and you have what I wrote above. However, $\theta$ depends on your covariates $x$ and that is why they have $h_{\theta}(x)$. In the same way, you can also write $\pi$ as $\pi(x)$ in the notation I used and you'd have the same description. By the way, notice I fixed a typo a moment ago. $\endgroup$
    – r_31415
    Feb 15, 2015 at 17:53
  • $\begingroup$ Another change I did was to use Bernoulli distributions to model the binary outcomes in the logistic regression case, since that is what you meant based on your references. $\endgroup$
    – r_31415
    Feb 15, 2015 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.