Which definition of Likelihood function is correct?

Question

In the online version of the Deep Learning book on chapter 5 the estimator for likelihood function is defined as:

That is the product of individual probabilities. After taking the log it arrives at the log-likelihood funciton (Eq.1):

It then rescales the above by dividing it by m to obtain a version expressed as the expected value (Eq.2):

OK. Here is my first question: The expected value is defined as

And so I think the expected value of log will be the below expression which is not the same as Eq.1 in the book.

The only way for this to be true(that is to divide the Eq.1 by m and arrive at Eq.2 as claimed by the book author) is for the probabilities of p(data) to be uniform. But this is also not a valid assumption. So I don't see what I am missing here?

---

Next, the book argues that maximizing the above log-likelihood function (Eq.2) is same as minimizing the KL divergence:

Or more simply just minimizing the second term. And so the author says that either way we arrive at the same function as Eq.2.

On the other hand, from the Wikipedia page the cross entropy of two probability is defined as :

I can understand this definition as the expected value of log(q) which is same as the expression in the Eq.2. but not Eq.1

From the same Wiki page the likelihood definition is given as below which is different than the likelihood function definition from the book(above). Here the probability of q (model) has been raised to the number of occurrences; which then on taking log it is understandable to see it as the expected value.

So I am confused that first of all which definition of likelihood function is the correct one? Given the definition from the Wikipedia I can understand that maximizing the log-likelihood function is same as minimizing the cross-entropy function. However I cannot arrive at the same conclusion from the definition of the likelihood and the log-likelihood function given in the book, for the reasons I explained above.

Answer 1

1 - Equivalence between $\mathbb{E}_{x\sim \hat{p}_{data}}\log(p_{model}(x;\theta))$ and $\frac{1}{m}\sum_{i=1}^m \log(p_{model}(x;\theta))$:

Yes, as you mentioned in the question, the key issue here is to realize that the following expectation is with respect to the empirical distribution defined by the training data ($\hat{p}_{data}$): $$ \theta_{ML} = \arg\max_{\theta}\mathbb{E}_{x\sim \hat{p}_{data}} \log(p_{model}(x;\theta)) $$ By definition, an empirical distribution assigns a probability of $1/m$ on each of the $m$ points $\mathbb{X} = \{x^{(1)},x^{(2)},...,x^{(m)}\}$ $\to$ Thereby the value of $\hat{p}_{data}(x_i)=\frac{1}{m}f(x_i)$ where $f(x_i)$ represents the ocurrence frequency of a value $x_i$ in our dataset.

Note that a value $x_i$ may appear more than once in our dataset $\mathbb{X}$ $\Rightarrow f(x_i)$ can be $> 1$. This is why we can express the previous expectation as: $$\begin{align} \mathbb{E}_{x\sim \hat{p}_{data}} \log(p_{model}(x;\theta)) &= \sum_{i=1}^m \frac{1}{m}\log(p_{model}(x^{(i)};\theta))\\ \end{align}$$ Hence, the maximization of both expressions lead to the same value of $\theta_{ML}$

---

2 - Equivalence between the minimization of $D_{KL}(\hat{p}_{data} \Vert p_{model})$ and $H(\hat{p}_{data}, p_{model})$

As said in the question, the KL divergence of the probability distributions $\hat{p}_{data}$ (empirical distribution of $\mathbb{X}$) and $p_{model}$ (our statistical model that we are using to fit $\mathbb{X}$) is given by the next expression: $$D_{\text{KL}}(\hat{p}_{data} \parallel p_{model}) = \mathbb{E}_{x\sim \hat{p}_{data}} [ \log(\hat{p}_{data}(x)) - \log(p_{model}(x;\theta))]$$

Here, we can see that $\mathbb{E}_{x\sim \hat{p}_{data}} \log(\hat{p}_{data}(x))$ does not depend on $\theta$ (it only depends on the data generating process), so it can be trated as a constant. Hence we can adress the same problem of finding the value of $\theta$ that minimizes the cross entropy by minimizing this KL divergence, because this is the same as minimizing:

$$ \mathbb{E}_{x\sim \hat{p}_{data}} [- \log(p_{model}(x;\theta))] = - \sum_{i=1}^m \frac{1}{m}\log(p_{model}(x^{(i)};\theta))$$

Which is the definition of the cross entropy $H(\hat{p}_{data}, p_{model})$.

---

3 - Equivalence between the minimization of $H(\hat{p}_{data}, p_{model})$ and the maximization of $\sum_{i=1}^m \log(p_{model}(x;\theta))$

Here, we have to proof that $$\arg\max_{\theta}\sum_{i=1}^m \log(p_{model}(x;\theta))$$ gives the same value of $\theta_{ML}$ than: $$\begin{align} \arg\min_{\theta}H(\hat{p}_{data}, p_{model}) &= \arg\min_{\theta}-\frac{1}{m}\sum_{i=1}^m \log(p_{model}(x^{(i)};\theta))\\ &= \arg\max_{\theta} \frac{1}{m}\sum_{i=1}^m \log(p_{model}(x^{(i)};\theta)) \end{align}$$

Both expressions are equivalent (they give the same value of $\theta_{ML}$) because a scale factor (such as $1/m$) does not affect in a minimization/ maximization problem.

---

4 - Equivalence between $\prod_{i=1}^m p_{model}(x^{(i)};\theta)$ and $\prod_{j} p_{model}(x_j;\theta)^{\#\text{ ocurrences of }j}$

This is because $i$ and $j$ represent different things:

• $i$ represents each individual datapoint of a dataset $\mathbb{X}\Rightarrow x^{(i)}$ is a datapoint.
• $j$ represents a certain value present in our dataset $\mathbb{X}\Rightarrow x_j$ is a certain value.

Given this, if a datapoint $x^{(i)}$ has the same value ($x_j$) of other $n$ datapoints, then we can express the likelihood of these datapoints in two equivalent ways: $$ \prod_{i=1}^n p_{model}(x^{(i)};\theta) = p_{model}(x_j;\theta)^n$$

So, if we extend this reasoning to all the possible values ($x_j$) that our dataset can have, we reach the equivalency given by wikipedia: $$\prod_{i=1}^m p_{model}(x^{(i)};\theta) = \prod_{j} p_{model}(x_j;\theta)^{\#\text{ ocurrences of }j}$$