0
$\begingroup$

I am confused about the negative log likelihood of a guassian model:

enter image description here

Can the negative likelihood of a gaussian model be negative ? lets suppose that the variance is going to 0 faster than the MSE the first term (log \sigma(xi)^2) will go to minus infinity.

What am I missing here ?

$\endgroup$

1 Answer 1

1
$\begingroup$

You aren't missing anything. The negative log-likelihood of any distribution can be negative.

In the case of the negative log-likelihood for a Gaussian random variable, this occurs when the function is evaluated at a particular $y_i$ that is highly probable given $\mu(x_i)$ and $\sigma(x_i)$, i.e. when $\mu(x_i)$ is reasonably close to $y_i$ and $\sigma(x_i)$ is sufficiently small. Positive values occur in the opposite case, when the (fitted) model described by $\mu(x_i)$ and $\sigma(x_i)$ gives low density to $y_i$ (i.e. a poor probabilistic prediction).

Note that in maximum likelihood estimation, the goal is to maximize the likelihood function, not the negative likelihood function. If you throw a negative sign in front, of course this switches the objective to minimizing, in which case, highly negative values of the negative log-likelihood are preferred.

Your observation about the function possibly tending to infinity is also highly relevant; in fact, this is why the log scoring rule (often called ELPD in Bayesian statistics) can behave quite unstable in practice.

$\endgroup$
1
  • 1
    $\begingroup$ thank you very much for this insight ! $\endgroup$
    – yoshcn
    Jan 1 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.