How to interpreter Binary Cross Entropy loss function?

Question

I saw some examples of Autoencoders (on images) which use sigmoid as output layer and BinaryCrossentropy as loss function.

The input to the Autoencoders is normalized [0..1] The sigmoid outputs values (value of each pixel of the image) [0..1]

I tried to evaluate the output of BinaryCrossentropy and I'm confused.

Assume for simplicity we have image [2x2] and we run Autoencoder and get 2 results. One result is close to the True value and the second is same as the true value:

import numpy as np
import tensorflow as tf

bce = tf.keras.losses.BinaryCrossentropy()

y_true = [0.5, 0.3, 0.5, 0.9]
y_pred = [0.1, 0.3, 0.5, 0.8]
print(bce(y_true, y_pred).numpy())

y_pred = [0.5, 0.3, 0.5, 0.9]
print(bce(y_true, y_pred).numpy())

Results:

0.71743906
0.5805602

As you can see, the second example (which is the same as the true value) gets low score (low loss value, but still it's not 0 or close to 0).

It seems that

It seems that using BinaryCrossentropy as loss function won't give us the best results. (We never get values close to zero) ?

Does the best value will be close to 0.5 ?

What am I missing ?

Answer 1

Binary cross entropy loss assumes that the values you are trying to predict are either 0 and 1, and not continuous between 0 and 1 as in your example. Because of this even if the predicted values are equal to the actual values your loss will not be equal to 0. Using values of either 0 or 1 does return a loss of zero in the case that the predicted values equal the true values:

import torch
from torch.nn import BCELoss

loss = BCELoss()

true = torch.Tensor([0.5, 0.3, 0.5, 0.9])
pred = torch.Tensor([0.5, 0.3, 0.5, 0.9])

loss(true, pred)
# tensor(0.5806)

true = torch.Tensor([1, 0, 1, 1])
pred = torch.Tensor([1, 0, 1, 1])

loss(true, pred)
# tensor(0.)

Answer 2

Binary cross entropy is intended to be used with data that take values in $\{0,1\}$ (hence binary). The loss function is given by, $$ \mathcal{L}_n = - \left[ y_n \cdot \log \sigma(x_n) + (1 - y_n) \cdot \log (1 - \sigma(x_n)) \right]$$ for a single sample $n$ (taken from Pytorch documentation) where $\sigma(x_n)$ is the predicted output.

For $y_n=0$ or $y_n=1$, the loss function as a function of $\sigma(x_n)$ is only 0 if $\sigma(x_n)=0$ or $\sigma(x_n)=1$, as you can see in the plot below. And although it's not what is intended with the binary cross entropy loss, you could in principle have a target value of $y_n=0.5$, and the loss would reach its minimum at $\sigma(x_n)=0.5$, though the loss would not be equal to 0.

In the plot below I show the loss function $\mathcal{L}(\sigma(x_n))$ for various values of the target $y_n$:

Answer 3

From your description you represent your [2x2] image as a vector of dimension 4 and then you compute binary cross entropy between y_pred and y_true.

1. First of all, cross entropy compares distribution $q$ relative to $p$, where $q$ is often used as the estimate and $p$ as ground truth in machine learning. $$ H(p, q) = -\text{E}_{p}[\log q] \geqslant H(p). $$ In this sense, $p,q$ are distributions and we don't need them to be binary. However, notice that the minimum value of cross entropy is the entropy of the ground truth. So, in your application, if you do not have binary target, you won't get zero BinaryCrossentropy even if your y_pred is identical to your y_true. Furthermore, your vectors represent the [2x2] image do not sum up to 1, so the above minimum cannot apply.

2. Why usual binary cross entropy has minimum $0$? The answer is that BinaryCrossentropy forces the ground truth $p$ to be binary, i.e. 0 or 1. In this case, the entropy $H(p)$ is zero so the minimum of $H(p,q)$ is zero.

In summary, answering your question,

1. you didn't apply cross entropy function correctly as you apply it to two vectors rather than two probability distributions.
2. if you do not have binary target values in your application, normalize the vector to get probability distributions and then apply cross entropy. However, in this case you will not get 0 even if your y_pred is completely the same as your y_true.