2
$\begingroup$

I'm a beginner in Machine learning and I've studied that collinearity among the predictor variables of a model is a huge problem since it can lead to unpredictable model behaviour and a large error. But, are there some models (say GLM) that are perhaps 'okay' with collinearity unlike the classic linear regression? It is said that the classic linear regression assumes there is no correlation between its independent variables.

This question arises because I was doing a project that said "If the input features are correlated with each other, it is better to use a generalized linear model since they would perform better than linear regression."

Can someone explain this ?

$\endgroup$
1
  • $\begingroup$ Better to use a GLM than to use what? Guessing the outcome? Modeling with a neural network? Support vector regression? $\endgroup$
    – Dave
    Commented Jun 20, 2020 at 15:03

2 Answers 2

5
$\begingroup$

“It is said that the classic linear regression assumes there is no correlation between its independent variables”

Depending on your goal of doing the regression, this common statement is false.

Even with multicollinearity, you get that $\hat{\beta}=(X^TX)^{-1}X^Ty$ is the solution to the OLS optimization.

Even with multicollinearity, you get that $\hat{\beta}=(X^TX)^{-1}X^Ty$ is minimum variance linear unbiased estimator from the Gauss-Markov theorem.

What the Gauss-Markov theorem requires is that the error terms not be correlated. This is commonly confused for saying that the predictors are not correlated, but that is indeed a mistake.

There can be numerical instability when you do math on a computer, particularly as you approach perfect multicollinearity ($X^TX$ is close to singular, singular in the extreme case of perfect multicollinearity or correlation of $1$ between variables), but there is no inherent problem with multicollinearity if your goal is to predict.

Where multicollinearity can hurt is when you want to do inference on the parameters, which is rarely a goal in machine learning. When you have multicollinearity, the parameter standard errors are inflated, sapping away your power to tell that they are not zero. Philosophically, it also makes it hard to attribute an effect to a particular predictor if it is correlated with others. (Imagine a hospital wanting to know if it pays its neurosurgeons as much as its heart surgeons and sees that the heart surgeons make way more but also sees that the heart surgeons have much more experience. Do they make more because of their specialty or because of their experience?)

Multicollinearity also can mean that you might be able to use a smaller amount of variables to get almost as much information as the entire set of variables. For instance, if two predictors are highly correlated, it might not be worth including both; you might be better off leaving out one for the sake of model parsimony and having fewer parameters in your regression, but that is an empirical problem and up to the model-designer’s judgment. I discuss in more detail here why it is not a given that dropping a variable will help you make accurate predictions.

Getting to the full GLM framework, the Gauss-Markov theorem does not apply, but the idea remains that there is no inherent issue with multicollinearity when your goal is to predict instead of to do parameter inference, which is the typical goal in machine learning.

$\endgroup$
3
  • 1
    $\begingroup$ But, if there are perfectly correlated variables, the matrix Xt X is not invertible, right? $\endgroup$ Commented Jul 18, 2020 at 6:58
  • $\begingroup$ @DavidMasip Yes, but if you have perfect correlation or perfect multicollinearity, then you have redundant information in your predictors and should address that; such an issue is not limited to linear regression. $\endgroup$
    – Dave
    Commented Jul 18, 2020 at 13:43
  • 1
    $\begingroup$ Ok, I think the case is a little bit different from other algorithms. For decision tree, you could have the same variable repeated 3 times the decision tree can be fitted perfectly, although it might not be optimal. For linear regression, the matrix cannot be inverted - we cannot fit via maximum likelihood. To me this is a bigger issue than in the decision tree case $\endgroup$ Commented Jul 18, 2020 at 14:40
0
$\begingroup$

Generalized Linear model is based on assumption of non-normal distribution and offer an alternative to the problem of response variable being measured as ,say, binomial data. In such a case,a model should handle non-linear relationship between predictor variables and dependent variable (along with constant or linear relationship between dependent measure and predictor variable).Several variants of Glm algorithms (link function) are possible that can handle non-linear effect of predictor variables on dependent categorical response variable. Multicollinearity does not seem to be an issue here.(Two variables are considered perfectly collinear if their correlation coefficient is +/- 1.0 and perfect collinearity needs to be understood in the context of correlation coefficient). GlM serves a different purpose and can not be compared with Linear Model/linear regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.