2
$\begingroup$

Cross-entropy (loss), $-\sum y_i\;\log(\hat{p_i})$, estimates the amount of information needed to encode $y$ using Huffman encoding based on the estimated probabilities $\hat{p}$. Therefore one could claim it should be considered to measure the amount of information, for example a number of bits.

Depending on the base of $\log$, these can be binary bits or digits, but typically are Euler-bits since $\ln$ is mostly used. Is there a popular or official name or unit for these so called Euler-bits? Is it OK to consider the unit of cross-entropy loss this way?

Obviously, loss is used for optimization and most don't care about the exact unit, but I'm curious and would like to use the correct terminology when explaining ML to others.

$\endgroup$
2
$\begingroup$

Basically, the first one who defines such amount of information was Shanon thanks to his definition of entropy: https://en.wikipedia.org/wiki/Entropy_(information_theory)

using bits of Shanon.

It describes the level of "interest" or "surprise" of information and it is also based on log 2. But I don't know if it is applicable to cross entropy.

$\endgroup$
1
  • 1
    $\begingroup$ So they are natural units or nats. Thanks. $\endgroup$
    – Herbert
    Jun 22 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.