Suppose for a single training example, the true label is [1 0 0 0 0] while the predictions be [0.1 0.5 0.1 0.1 0.2]. How to calculate its binary cross entropy? And how output value decides whether a sample belongs to class a or b? I also want to know that that in the formula of binary cross entropy what is 1-p?
1
-
$\begingroup$ Calculating cross entropy has been asked and explained many times (e.g. stackoverflow.com/questions/41990250/what-is-cross-entropy). Output class is the argmax of output values. 1-p means one minus p. $\endgroup$– user12075Sep 30, 2018 at 17:47
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Binary cross-entropy is a simplification of the cross-entropy loss function applied to cases where there are only two output classes. Essentially it can be boiled down to the negative log of the probability associated with your true class label.
Cross entropy
Cross entropy is defined as
$L = -\sum ylog(p)$
where $y$ is the binary class label, 1 if the correct class 0 otherwise. And $p$ is the probability of each class.
Let's look at an example, if for an instance $X$ the output label is 0 and your model output was $[0.7, 0.3]$. Then we can see that the loss function using binary cross entropy is
$L = -(1*log(0.7) + 0*log(0.3)) = 0.155$
Notice how the loss is only affected by the true label's prediction probability.
Binary cross-entrop
Binary cross entropy is for an example with 2 output classes and can be simplified as
$L = (ylog(p) + (1-y)log(1-p)) = (1*log(0.7) + (1-1)log(1-0.7)) = 0.155$
Your example
For your example, $[1,0,0,0,0]$ and predictions $[0.1 0.5 0.1 0.1 0.2]$ the loss is
$L = -(1*log(0.1) + 0*log(0.5) + 0*log(0.1) + 0*log(0.1) + 0*log(0.2)) = 1$
-
$\begingroup$ ok So what the higher value of L tells us? Does it gives any information about the two classes or about our model?I have to use this in machine learning $\endgroup$– herryOct 1, 2018 at 7:09