The connection between cross entropy and log likelihood is widely expressed for the case when sample multi-class labels are one hot binary vectors (basically the same). Cross entropy is defined when labels are not one-hot but any valid probability distribution.

What is the relation between the general cross entropy and log-likelihood?

One-hot case

Both log likelihood and cross entropy come down to:



$i$ - the sample index
$k$ - the class index
$\theta$ - model parameters
$y$ - samples labels (one-hot vector)
$\hat{y}$ - predictions (probability distribution like vector)
$1\{\}$ - indicator function

General case ($y_i$ is probability distribution)

Cross entropy can be reduced to:

How does this relate to log-likelihood?

  • 1
    $\begingroup$ Which log likelihood? Every dfistribution has a log likelihood. You should be concrete and post the equations you're describing here. I suspect that what you're referring to as "cross entropy" is the log likelihood of the multinomial distribution, but to be sure you should be concrete about both. $\endgroup$ – David Marx Feb 24 '18 at 12:02
  • $\begingroup$ @DavidMarx Added more details. I hope it makes things clearer. $\endgroup$ – Xyand Feb 24 '18 at 14:08

If I understand it correctly, you are asking how log likelihood in a multi-class classification problem relates to the cross entropy loss. So here is my try:

Assuming we have a multi class classification problem ($C$ different classes) where we estimated the conditional probabilities for each class given the data $x$ (e.g., using a neural network) and where the classes are one-hot encoded: $$ \hat{p}_i^{(k)} \in [0, 1] \quad \text{denotes the estimated probability of the } i\text{th sample for the } k\text{th class}\\ p_i^{(k)}\in \{0, 1\}\quad \text{denotes the true probability of the } i\text{th sample for the } k\text{th class}\\ y_i^{(k)} = p_i^{(k)} = 1_{y_{i}^{(k)}==1} \quad \text{hard labels, i.e.,} \begin{cases} 1 & \text{if }i \text{th sample belongs to } k\text{th class},\\ 0 &\text{else.}\end{cases} $$ Note that $\hat{p}_i^{(k)} = \hat{p}_i^{(k)} (\textbf{x}_i, \theta)$ is actually a function that depends on parameters $\theta$ (e.g., network weights) and the input data $\textbf{x}_i$. In the end, we want to optimize the parameters $\theta$. For the sake of simplicity, we just write $\hat{p}_i^{(k)}$ (silently knowing that these estimated probabilities come from some kind of function approximator).

The idea of maximum likelihood is to maximize the likelihood of our estimated conditional probabilities given the data, i.e., $$ \large{ \max P(Y|X) = \max \left( \prod_{i=1}^N P(y_i| x_i) \right) =\max \left( \prod_{i=1}^N \prod_{k=1}^C \hspace{0.2cm} \left(\hat{p}_i^{(k)} \right)^{1_{y_i^{(k)} == 1}} \right)} $$ Let's take a moment to digest this formula. It's actually pretty easy what we are doing here, we want that the estimated conditional probabilities $\hat{p}_i^{(k)} \in [0, 1]$ correspond to the true labels $p_i^{(k)}\in \{0, 1\}$. Note that the true labels are hard labels, e.g., $\textbf{p}_0 = \begin{bmatrix}0 &1&0 \end{bmatrix}^{\text{T}}$, but the predictions are (probably) soft labels, e.g., $\hat{\textbf{p}}_0 = \begin{bmatrix}0.2 &0.7&0.1 \end{bmatrix}^{\text{T}}$. We optimize our likelihood by maximizing the predicted probability of the true label, i.e., where the label $y_i$ corresponds to the class $k$, this is denoted by the indicator function $1_{y_i^{(k)} == 1}$

In the extreme case where our conditional probabilities always predict the true class with probability 1, this yields the maximum possible value for our likelihood: $$ \max P(Y|X) = 1 $$

By definition of the hard labels, we can rewrite the formula of the likelihood into $$ P(Y|X) = \prod_{i=1}^N \prod _{k=1}^C {\left(\hat{p}_i^{(k)}\right)}^{p_i^{(k)}} $$

Now we take the negative log-likelihood (hence the maximization problem becomes a minimization problem): $$ \max P(Y|X) \equiv \min \left(- \log P(Y|X)\right) = \min \left(- \sum_{i=1}^N \sum_{k=1}^C p_i^{(k)} \cdot \log \left(\hat{p}_i^{(k)}\right) \right) $$ (This formula is the same as in your question).

Let's suppose, there is only one datapoint $N=1$, then we get $$ - \log P(Y|X) = - \sum_{k=1}^C p^{(k)} \cdot \log \left(\hat{p}^{(k)}\right) = H(p, \hat{p}) $$ This is known as the cross-entropy loss, i.e., minimization of the cross-entropy loss corresponds to maximum likelihood when hard labels are provided.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.