10
$\begingroup$

I have a data set which has continuous independent variables and a continuous dependent variable. To predict the dependent variable using the independent variables, I've run an ensemble of regression models and tried to compare them against each other. Here are the results for reference:

enter image description here

I can interpret what the R-squared value / Coefficient of determination for each of those models means. However, I can't understand what the Negative Log Likelihood means. Especially, why is it Infinity for Linear Regression and Boosted Decision Tree, and a finite value for a Decision Forest Regression?

Edit:
Data Description: The data that went into these three models is all continuous independent variables and a continuous dependent variable. There are a total of 542 observations and 26 variables.
These 542 variables are split 70 - 30 to get training and testing datasets. Therefore, the training dataset has 379 observations and 26 variables; the testing dataset has 163 observations and 26 variables. No missing data.

Edit 2 Possible Explanation - (click here): Apparently, Linear Regression and Boosted Trees in Azure ML don't calculate the Negative Log-Likelihood metric - and that could be the reason that NLL is infinity or undefined in both cases.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Does that "rows: 3 columns:6" mean you have three data points and six explanatory variables? That's going to mess up a linear regression because you are trying to find more variables than you have data points. So I suspect -log(Lik) is Inf because of that.... If -log(Lik)=Inf then log(Lik)=-Inf then Lik=0 - somehow the likelihood is zero.... $\endgroup$
    – Spacedman
    Sep 2, 2016 at 22:05
  • $\begingroup$ @Spacedman I've added my guess, please take a look. $\endgroup$
    – ABCD
    Sep 3, 2016 at 10:46
  • $\begingroup$ @Spacedman: I expect that "rows:3 columns: 6" is referring to the displayed table - which has 3 rows and 6 columns - and is not a description of the input data. It's just a generic table display widget that happens to be showing the summary data for the question. $\endgroup$
    – Neil Slater
    Sep 3, 2016 at 11:25
  • $\begingroup$ @minu You need to give us some more info about your data set and the model specification because I suspect something is degenerate about it and its only possible to guess wildly without knowing both. $\endgroup$
    – Spacedman
    Sep 3, 2016 at 11:59
  • $\begingroup$ 3 rows and 6 columns refers to the displayed table of evaluation results. Please find edits in my question with a little more data description. $\endgroup$
    – Minu
    Sep 3, 2016 at 15:55

2 Answers 2

Reset to default
7
$\begingroup$

Likelihood function is the product of probability distribution function, assuming each observation is independent. However, we usually work on a logarithmic scale, because the PDF terms are now additive. If you don't understand what I've said, just remember the higher the value it is, the more likely your model fits the model. Google for maximum likelihood estimation if you're interested.

Obviously, your input data is bad. You should give your model a proper data set. While I don't have your data set, we can take a look at the likelihood function for linear regression:

enter image description here

You will get infinity if the likelihood function is zero or undefined (that's because log(0) is invalid). Look at the equation, most likely your sample standard deviation is zero. If it's zero, the last term will be undefined. Have you given a data set that you copied and pasted the same data over rows?

Boosted trees should also be undefined if your sample deviation is zero. However, decision tree is estimated based on impurity and won't crash here.

Summary: please check and double check your data.

EDIT I think you just had a bug. Linear regression will always give you something here. Have you fitted the models in R with the same dataset? –

Improve this answer
$\endgroup$
4
  • $\begingroup$ I've added some data description in the question. Is this happening because my observations are too less for the spread (variables) in my data? I haven't copied and pasted the same data over rows. What can I do to make this data viable for modeling? $\endgroup$
    – Minu
    Sep 3, 2016 at 16:10
  • $\begingroup$ I've checked the standard deviation for each of the 26 variables, and none of those are 0. I'll try R. $\endgroup$
    – Minu
    Sep 4, 2016 at 14:27
  • $\begingroup$ Thanks for your answer @StudentT. Helped me figure out what NLL means, and why it's important. But, apparently, Linear Regression and Boosted Trees in Azure ML don't calculate the NLL metric - and that could be the reason that NLL is infinity or undefined in both cases. Do you have any comments on the NLL value of the Decision Forest? $\endgroup$
    – Minu
    Sep 6, 2016 at 12:53
  • $\begingroup$ "the more likely your model fits the model", could you explain what you mean by this please? $\endgroup$
    – baxx
    Mar 22, 2019 at 10:49
5
$\begingroup$

This answer correctly explains how the likelihood describes how likely it is to observe the ground truth labels t with the given data x and the learned weights w. But that answer did not explain the negative.

$$ arg\: max_{\mathbf{w}} \; log(p(\mathbf{t} | \mathbf{x}, \mathbf{w})) $$

Of course we choose the weights w that maximize the probability.

But to optimize it, we need a minimum function that we set to zero to get the local/global minimum.

That's why instead of maximizing the function we minimize its negative:

$$ arg\: min_{\mathbf{w}} \; -log(p(\mathbf{t} | \mathbf{x}, \mathbf{w})) $$

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.