Consider 2 regression models on the same data set.
Model 1 : $R^2 = 90$% , $R^2(adjusted) = 80 $%, $R^2(pred) = 70$%
Model 2 : $R^2 = 60$% , $R^2(adjusted) = 59 $%, $R^2(pred) = 58$%
In the first model , all values are very high but the difference between them is also very high.
In model 2, the values are low but difference between them is less.
Which of the above 2 models is better and why?


Which one of the two models is better?

It depends what you care about.


If you just want the highest $R^2$, $R_{adj.}$ or $R_{pred}$, obviously Model 1 is better. In fact, in your example, all performance metrics would tell you to go for Model 1! But there are other aspects to picking a model...

Many people care about the density of explanability (I made that term up); think of it as the explanatory power normalised by number of degrees of freedom (also referred to as the elegance of the model). If that was your preference, then Model 2 look more attractive, as its metrics drop by only 1% and 1% again, compared to Model 1's 10% and 10% drop.

Some relevant background

There is the term Minimum Description Length, which formalises Ockham's Razor, stating:

the best hypothesis (a model and its parameters) for a given set of data is the one that leads to the best compression of the data.

This can help decide which of two models is best, by assessing how dense the explanability of each model. In simple terms: assume models A and B both produce 80% accuracy, but A uses an extra parameter, then B is the more elegant model as it uses its input more efficiently. In other words, its compression of the information is higher.

Here is a really nice blog post (published today!) that explains how Ockham's Razor, Bayes' and Shannon Entropy all come together to equally define how thie density of information in a model is a really important criterion. Spoiler: all three boil down to the same guiding principal.

Another way people try to decide which regression model is best, is to use another metric such as the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC), both of which try to find a compromise between the performance of a model and the number of degrees of freedom (i.e. number of parameters) that were used. We pick the model with the lowest score. For example, the AIC is defined as:

$$ AIC = 2k-2 ln({\hat {L}}), $$

where $k$ is the number of parameters and $\hat{L}$ is the performance metric (here the Maximum Likelihood is implied). So in plain English, the AIC gets larger and alrger as we include more parameters and out model becomes more complex. However, if the model fits the data more accuractely with each parameter, that is subtracted and so reduces the AIC value. The BIC essentially works the same way. I think it is intuitive what we are trying to do here.


Coming back to my first answer: "It depends what you care about" - it hopefully now clear that the selection is subjective. If you can live with high model complexity, you might just go for the best $R_{pred.}$ and so Model 1. On the other hand, if you enjoy an elegant model with high information density, the deltas of 1% between your three metrics will lead you to select Model 2.


A term I was searching for came back to me: parsimony. A parsimonious model is one that reaches accuracy expectation and does so with a few parameters as possible. It is all about maximum information density and explanatory efficiency. Check out this thread that is relevant to your questions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.