What Base Should Be Used For Negative Log Likelihood?

Question

When calculating the negative log likelihood loss, what base of log are we supposed to use?

Answer 1

Typically it is implemented as the natural logarithm, base e. Other bases can be used for the same effect though.

Answer 2

The change in base is equivalent to multiplying the function by a constant. It does not affect the computation.

$ log_b(x) = \dfrac{1}{log_e(b)}.log_e(x) $

Answer 3

Generally, when the log likelihood is being calculated, it's being done as a loss function, that is, an amount that is being optimized. Changing the base multiplies the log by a constant. As long as the bases are either both greater than one, or both less than one, this constant is positive (note that "negative log likelihood" can be interpreted as taking the log base a number less than one), and multiplying a function by a constant greater than one doesn't affect what inputs optimize the value of that function. In other words, it doesn't matter. Changing the base basically is a change of units: the log base $2$ is units of bits, log base $256$ is units of bytes, log base $e$ is units of nits. So it's like asking "Okay, we're trying to minimize the amount of wire that we're using ... but are we minimizing the amount of wire in feet, or the amount of wire in meters?"

The natural base $e$ is often used because it makes some of the math easier, but the base $2$ is also used in some contexts because it allows reporting the log in the units of bits. In cases where the absolute, rather relative, value of log likelihood is important, the base should be indicated either by explicitly naming the base or giving the units (e.g. bits, nits, etc.).