Is Categorical Crossentropy always bounded between 0 and 1, or is it possible that during training of a Neural Network it can get higher values?

More specifically, I'm referring to the TensorFlow 2.0 function.


You can actually get values from $[0, \infty[$. Consider the cross entropy loss be following formula:

$loss (y_{pred}, y_{true}) = - \log{y_{pred, class_i}}$

, whereas the index $class_i$ states that you only use the output of the $i$th class.

For example think that your true class is $dog$, which is represented by the $3$rd output neuron. On the $3$rd output neuron your network outputs a probability of $10\%$. Your loss is then $-\log{10\%}= 2.3$. Further if your probability is only $0.1\%$ your loss will equal $6.9$ and so on.

Remember that:

$\lim\limits_{x \rightarrow 0}{\log{x}} = -\infty$

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.