# What Base Should Be Used For Negative Log Likelihood?

When calculating the negative log likelihood loss, what base of log are we supposed to use?

Typically it is implemented as the natural logarithm, base e. Other bases can be used for the same effect though.

• (+1) for "Typically it is implemented as the natural logarithm, base e. Other bases can be used". However "for the same effect" may be slightly misleading -- there's a reason the natural logarithm is usually used: For many distributions, it makes the math convenient. Using some other base, while convenient in some cases, would not be as convenient as often as the natural logarithm. – duckmayr Nov 3 '18 at 15:44

The change in base is equivalent to multiplying the function by a constant. It does not affect the computation.

$$log_b(x) = \dfrac{1}{log_e(b)}.log_e(x)$$

Generally, when the log likelihood is being calculated, it's being done as a loss function, that is, an amount that is being optimized. Changing the base multiplies the log by a constant. As long as the bases are either both greater than one, or both less than one, this constant is positive (note that "negative log likelihood" can be interpreted as taking the log base a number less than one), and multiplying a function by a constant greater than one doesn't affect what inputs optimize the value of that function. In other words, it doesn't matter. Changing the base basically is a change of units: the log base $$2$$ is units of bits, log base $$256$$ is units of bytes, log base $$e$$ is units of nits. So it's like asking "Okay, we're trying to minimize the amount of wire that we're using ... but are we minimizing the amount of wire in feet, or the amount of wire in meters?"

The natural base $$e$$ is often used because it makes some of the math easier, but the base $$2$$ is also used in some contexts because it allows reporting the log in the units of bits. In cases where the absolute, rather relative, value of log likelihood is important, the base should be indicated either by explicitly naming the base or giving the units (e.g. bits, nits, etc.).

• Why is bits/nits the correct unit to use here? As far as I understand, it is used for entropy, which is a different thing. I haven't seen it commonly used for log-likelihoods. – Federico Poloni Nov 3 '18 at 20:34