Negative log-likelihood not the same as cross-entropy?

Question

The negative log-likelihood $$ \sum_{i=1}^{m}\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta}) $$ can be multiplied by $\frac{1}{m}$ after which the law of large numbers can be used to get $$ \frac{1}{m} \sum_{i=1}^{m}\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta}) \rightarrow E_{}(\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta})) $$ as the sample size $m$ tends to infinity. This expectation is the "cross-entropy".

Now here comes my question: The book I am reading(Deep Learning by Goodfellow et al) mentions several attractive properties of using the negative log-likelihood(like consistency). But meanwhile, it also also uses cross-entropy directly as the loss function of maximum likelihood estimators: This doesn't make sense to me - to talk about negative log-likelihood and cross-entropy as being identical. It would make sense for me to talk about NLL as an approximation of the cross-entropy.

1. I mean, they give different results - so why use one over the other? This seems like a valid question when they do not give the same results and must thus also affects the performance. Like, I am only aware of neural networks that use cross-entropy and not ones that use NLL - how come?

2. Maybe cross-entropy even holds other properties than negative log-likelihood?

Answer 1

I think the issue is that while

$$ \frac{1}{m} \sum_{i=1}^{m}\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta}) \rightarrow E_{}(\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta})) $$

is true, the right side is not what is being referred to as the cross-entropy. Indeed, we have no access to the expectation over the true population distribution / data generating process; a few paragraphs down, we find:

We can thus see maximum likelihood as an attempt to make the model distribution match the empirical distribution $\hat{p}_{data}$. Ideally, we would like to match the true data-generating distribution $p_{data}$, but we have no direct access to this distribution.

They also distinguish throughout that the expectations are over $\hat{p}_{data}$, the training data empirical distribution, not $p_{data}$, the underlying population distribution.

The left side of your limit is already an expectation, but over the finite training data, and that is what is referred to as the cross-entropy. ("Cross-entropy" is a broader term, for any pair of probability distributions. Goodfellow et al note this (my emphasis):

Any loss consisting of a negative log-likelihood is a cross-entropy between the empirical distribution defined by the training set and the probability distribution defined by model.

---

So, the answer to your questions is that the premise is incorrect: (this) cross-entropy is the same as negative log-likelihood. Taking your questions with the limiting and population cross-entropy instead, the answer is "we don't have access to the latter". It would be the better target to be sure, but our lack of that information is the point of modeling in the first place.