In the online version of the Deep Learning book on chapter 5 the estimator for likelihood function is defined as:

enter image description here

That is the product of individual probabilities. After taking the log it arrives at the log-likelihood funciton (Eq.1):

enter image description here

It then rescales the above by dividing it by m to obtain a version expressed as the expected value (Eq.2):

enter image description here

OK. Here is my first question: The expected value is defined as enter image description here

And so I think the expected value of log will be the below expression which is not the same as Eq.1 in the book.

enter image description here

The only way for this to be true(that is to divide the Eq.1 by m and arrive at Eq.2 as claimed by the book author) is for the probabilities of p(data) to be uniform. But this is also not a valid assumption. So I don't see what I am missing here?

Next, the book argues that maximizing the above log-likelihood function (Eq.2) is same as minimizing the KL divergence:

enter image description here

Or more simply just minimizing the second term. And so the author says that either way we arrive at the same function as Eq.2.

enter image description here

On the other hand, from the Wikipedia page the cross entropy of two probability is defined as :

enter image description here

I can understand this definition as the expected value of log(q) which is same as the expression in the Eq.2. but not Eq.1

From the same Wiki page the likelihood definition is given as below which is different than the likelihood function definition from the book(above). Here the probability of q (model) has been raised to the number of occurrences; which then on taking log it is understandable to see it as the expected value.

enter image description here

So I am confused that first of all which definition of likelihood function is the correct one? Given the definition from the Wikipedia I can understand that maximizing the log-likelihood function is same as minimizing the cross-entropy function. However I cannot arrive at the same conclusion from the definition of the likelihood and the log-likelihood function given in the book, for the reasons I explained above.


1 Answer 1


1 - Equivalence between $\mathbb{E}_{x\sim \hat{p}_{data}}\log(p_{model}(x;\theta))$ and $\frac{1}{m}\sum_{i=1}^m \log(p_{model}(x;\theta))$:

Yes, as you mentioned in the question, the key issue here is to realize that the following expectation is with respect to the empirical distribution defined by the training data ($\hat{p}_{data}$): $$ \theta_{ML} = \arg\max_{\theta}\mathbb{E}_{x\sim \hat{p}_{data}} \log(p_{model}(x;\theta)) $$ By definition, an empirical distribution assigns a probability of $1/m$ on each of the $m$ points $\mathbb{X} = \{x^{(1)},x^{(2)},...,x^{(m)}\}$ $\to$ Thereby the value of $\hat{p}_{data}(x_i)=\frac{1}{m}f(x_i)$ where $f(x_i)$ represents the ocurrence frequency of a value $x_i$ in our dataset.

Note that a value $x_i$ may appear more than once in our dataset $\mathbb{X}$ $\Rightarrow f(x_i)$ can be $> 1$. This is why we can express the previous expectation as: $$\begin{align} \mathbb{E}_{x\sim \hat{p}_{data}} \log(p_{model}(x;\theta)) &= \sum_{i=1}^m \frac{1}{m}\log(p_{model}(x^{(i)};\theta))\\ \end{align}$$ Hence, the maximization of both expressions lead to the same value of $\theta_{ML}$

2 - Equivalence between the minimization of $D_{KL}(\hat{p}_{data} \Vert p_{model})$ and $H(\hat{p}_{data}, p_{model})$

As said in the question, the KL divergence of the probability distributions $\hat{p}_{data}$ (empirical distribution of $\mathbb{X}$) and $p_{model}$ (our statistical model that we are using to fit $\mathbb{X}$) is given by the next expression: $$D_{\text{KL}}(\hat{p}_{data} \parallel p_{model}) = \mathbb{E}_{x\sim \hat{p}_{data}} [ \log(\hat{p}_{data}(x)) - \log(p_{model}(x;\theta))]$$

Here, we can see that $\mathbb{E}_{x\sim \hat{p}_{data}} \log(\hat{p}_{data}(x))$ does not depend on $\theta$ (it only depends on the data generating process), so it can be trated as a constant. Hence we can adress the same problem of finding the value of $\theta$ that minimizes the cross entropy by minimizing this KL divergence, because this is the same as minimizing:

$$ \mathbb{E}_{x\sim \hat{p}_{data}} [- \log(p_{model}(x;\theta))] = - \sum_{i=1}^m \frac{1}{m}\log(p_{model}(x^{(i)};\theta))$$

Which is the definition of the cross entropy $H(\hat{p}_{data}, p_{model})$.

3 - Equivalence between the minimization of $H(\hat{p}_{data}, p_{model})$ and the maximization of $\sum_{i=1}^m \log(p_{model}(x;\theta))$

Here, we have to proof that $$\arg\max_{\theta}\sum_{i=1}^m \log(p_{model}(x;\theta))$$ gives the same value of $\theta_{ML}$ than: $$\begin{align} \arg\min_{\theta}H(\hat{p}_{data}, p_{model}) &= \arg\min_{\theta}-\frac{1}{m}\sum_{i=1}^m \log(p_{model}(x^{(i)};\theta))\\ &= \arg\max_{\theta} \frac{1}{m}\sum_{i=1}^m \log(p_{model}(x^{(i)};\theta)) \end{align}$$

Both expressions are equivalent (they give the same value of $\theta_{ML}$) because a scale factor (such as $1/m$) does not affect in a minimization/ maximization problem.

4 - Equivalence between $\prod_{i=1}^m p_{model}(x^{(i)};\theta)$ and $\prod_{j} p_{model}(x_j;\theta)^{\#\text{ ocurrences of }j}$

This is because $i$ and $j$ represent different things:

  • $i$ represents each individual datapoint of a dataset $\mathbb{X}\Rightarrow x^{(i)}$ is a datapoint.
  • $j$ represents a certain value present in our dataset $\mathbb{X}\Rightarrow x_j$ is a certain value.

Given this, if a datapoint $x^{(i)}$ has the same value ($x_j$) of other $n$ datapoints, then we can express the likelihood of these datapoints in two equivalent ways: $$ \prod_{i=1}^n p_{model}(x^{(i)};\theta) = p_{model}(x_j;\theta)^n$$

So, if we extend this reasoning to all the possible values ($x_j$) that our dataset can have, we reach the equivalency given by wikipedia: $$\prod_{i=1}^m p_{model}(x^{(i)};\theta) = \prod_{j} p_{model}(x_j;\theta)^{\#\text{ ocurrences of }j}$$

  • $\begingroup$ That was perfect, thanks. But I still have couple of questions. if p(xi)=1/m f(xi), did you not miss multiplying 1/m by f(xi) in your definition of log likelihood ? $\endgroup$
    – ali
    Commented Dec 19, 2020 at 21:24
  • $\begingroup$ Sure, to what part of the answer are you refering to? $\endgroup$
    – Javier TG
    Commented Dec 19, 2020 at 21:28
  • $\begingroup$ in part 1, Ex∼p^data log(pmodel(x;θ))=∑1/m log(p model(x;θ)) $\endgroup$
    – ali
    Commented Dec 19, 2020 at 21:30
  • $\begingroup$ That's because the summatory is over the datapoints ($x^{(i)}$) and not over the values ($x_j$) (following the notation of part 4) present in our dataset $\mathbb{X}$. Thereby, the datapoints that have the same value are sumed and this has the same effect as multiplying a certain value of $\log(p_{model}(x_j;\theta))$ by $f(x_j)$. $\endgroup$
    – Javier TG
    Commented Dec 19, 2020 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.