The negative log-likelihood $$ \sum_{i=1}^{m}\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta}) $$ can be multiplied by $\frac{1}{m}$ after which the law of large numbers can be used to get $$ \frac{1}{m} \sum_{i=1}^{m}\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta}) \rightarrow E_{}(\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta})) $$ as the sample size $m$ tends to infinity. This expectation is the "cross-entropy".

Now here comes my question: The book I am reading(Deep Learning by Goodfellow et al) mentions several attractive properties of using the negative log-likelihood(like consistency). But meanwhile, it also also uses cross-entropy directly as the loss function of maximum likelihood estimators: enter image description here This doesn't make sense to me - to talk about negative log-likelihood and cross-entropy as being identical. It would make sense for me to talk about NLL as an approximation of the cross-entropy.

  1. I mean, they give different results - so why use one over the other? This seems like a valid question when they do not give the same results and must thus also affects the performance. Like, I am only aware of neural networks that use cross-entropy and not ones that use NLL - how come?

  2. Maybe cross-entropy even holds other properties than negative log-likelihood?

  • $\begingroup$ I am also very interested in this :) $\endgroup$
    – Ukula Udan
    May 7 '21 at 15:10
  • 1
    $\begingroup$ The excerpt being referenced can be found on page 130: deeplearningbook.org/contents/ml.html#pf23 $\endgroup$ May 7 '21 at 19:06
  • $\begingroup$ @BenReiniger If you can help me with the follow-up questions that I put as comments to your answer, you have helped so perfectly and of course I will hit that "accept" button or whatever its called :)) $\endgroup$
    – That Guy
    May 8 '21 at 20:34

I think the issue is that while

$$ \frac{1}{m} \sum_{i=1}^{m}\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta}) \rightarrow E_{}(\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta})) $$

is true, the right side is not what is being referred to as the cross-entropy. Indeed, we have no access to the expectation over the true population distribution / data generating process; a few paragraphs down, we find:

We can thus see maximum likelihood as an attempt to make the model distribution match the empirical distribution $\hat{p}_{data}$. Ideally, we would like to match the true data-generating distribution $p_{data}$, but we have no direct access to this distribution.

They also distinguish throughout that the expectations are over $\hat{p}_{data}$, the training data empirical distribution, not $p_{data}$, the underlying population distribution.

The left side of your limit is already an expectation, but over the finite training data, and that is what is referred to as the cross-entropy. ("Cross-entropy" is a broader term, for any pair of probability distributions. Goodfellow et al note this (my emphasis):

Any loss consisting of a negative log-likelihood is a cross-entropy between the empirical distribution defined by the training set and the probability distribution defined by model.

So, the answer to your questions is that the premise is incorrect: (this) cross-entropy is the same as negative log-likelihood. Taking your questions with the limiting and population cross-entropy instead, the answer is "we don't have access to the latter". It would be the better target to be sure, but our lack of that information is the point of modeling in the first place.

  • 1
    $\begingroup$ Thank you so much for your input! I really appreciate your time. I have some questions about your answer: 1) About the cross-entropy not be referred to by me correctly - I see that, thank you :) 2) You write "The left side of your limit is already an expectation, but over the finite training data, and that is what is referred to as the cross-entropy" - with the broader definition of cross-entropy with two distributions as its arguments, I see. But it is not the cross-entropy w.r.t. the empirical distribution and the model distribution, correct? $\endgroup$
    – That Guy
    May 7 '21 at 23:13
  • 1
    $\begingroup$ 1) Your concluding answer "So, the answer to your questions is..." that using the broader definition of cross-entropy that is not w.r.t. the empirical distribution and the model distribution(assuming you answer my question in the comment above)? 2) So an implementation that uses "cross-entropy" as the loss function, does it use the negative log-likelihood or the cross-entropy w.r.t. the emperical data distribution and the model distribution? I mean, it does have access to the emperical data distribution. PS: Can you refer to the individual questions here as "0"(the above), "1)", and "2)" ? $\endgroup$
    – That Guy
    May 7 '21 at 23:19
  • $\begingroup$ I really appreciate this <33 I ask these additional questions also because I believe it is rude if you put time into this and I do not make sure to extract all the goodies from it :)))) $\endgroup$
    – That Guy
    May 7 '21 at 23:20
  • 1
    $\begingroup$ Ahh, I think I am starting to get it! But you write "and in the empirical distribution, each sample is equally likely" - how is that true? With my enhanced understanding, $\hat{p}_{data}(\mathbf{y}_i | \mathbf{x}_i)$ is $1$ for $\mathbf{y}_i$ and $0$ for all the other possible labels of samples. In that case, each sample(when looking at a "sample" as being a label for a certain sample) is not equally likely? $\endgroup$
    – That Guy
    May 13 '21 at 14:48
  • 1
    $\begingroup$ Right, that's my fault for being sloppy about the conditionals. I was focusing on the textbook section in question, which doesn't discuss conditional probabilities (only very briefly adding that in the next section). I think the "conditional cross-entropy" isn't well described in the wild, but see stats.stackexchange.com/q/428937/232706 for a start. $\endgroup$ May 13 '21 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.