0
$\begingroup$

When calculating the negative log likelihood loss, what base of log are we supposed to use?

$\endgroup$
4
$\begingroup$

Typically it is implemented as the natural logarithm, base e. Other bases can be used for the same effect though.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ (+1) for "Typically it is implemented as the natural logarithm, base e. Other bases can be used". However "for the same effect" may be slightly misleading -- there's a reason the natural logarithm is usually used: For many distributions, it makes the math convenient. Using some other base, while convenient in some cases, would not be as convenient as often as the natural logarithm. $\endgroup$ – duckmayr Nov 3 '18 at 15:44
5
$\begingroup$

The change in base is equivalent to multiplying the function by a constant. It does not affect the computation.

$ log_b(x) = \dfrac{1}{log_e(b)}.log_e(x) $

| improve this answer | |
$\endgroup$
0
$\begingroup$

Generally, when the log likelihood is being calculated, it's being done as a loss function, that is, an amount that is being optimized. Changing the base multiplies the log by a constant. As long as the bases are either both greater than one, or both less than one, this constant is positive (note that "negative log likelihood" can be interpreted as taking the log base a number less than one), and multiplying a function by a constant greater than one doesn't affect what inputs optimize the value of that function. In other words, it doesn't matter. Changing the base basically is a change of units: the log base $2$ is units of bits, log base $256$ is units of bytes, log base $e$ is units of nits. So it's like asking "Okay, we're trying to minimize the amount of wire that we're using ... but are we minimizing the amount of wire in feet, or the amount of wire in meters?"

The natural base $e$ is often used because it makes some of the math easier, but the base $2$ is also used in some contexts because it allows reporting the log in the units of bits. In cases where the absolute, rather relative, value of log likelihood is important, the base should be indicated either by explicitly naming the base or giving the units (e.g. bits, nits, etc.).

| improve this answer | |
$\endgroup$
  • $\begingroup$ Why is bits/nits the correct unit to use here? As far as I understand, it is used for entropy, which is a different thing. I haven't seen it commonly used for log-likelihoods. $\endgroup$ – Federico Poloni Nov 3 '18 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.