7
$\begingroup$

Multicollinearity is a problem for linear regression because the results become unstable / depend too much on single elements (source).

(Also, the inverse of $X^TX$ doesn't exist so the standard OLS estimator does not exist ... I have no idea how, but sklearn deals with it just fine)

Is (perfect) multicollinearity also a problem for neural networks?

$\endgroup$
2
$\begingroup$

Multi colinearity affects the learning of Artificial Neural network. Since the information in the dependent variable is very less compared to the other variables, the neural network will take more time to converge.

In packages like sklearn, the dependent variables are identified and omitted from the calculation. I have used the lm function in R and it marks the coefficient of the dependent variable with NA. one can remove the variable from the calculation and still the coefficients are going to be same. In these cases, the rank of the x matrix will be less than the number of columns.

Even though there are no inverse exists for xTx, most of the packages will not calculate the inverse directly, but they will calculate the pseudo inverse.

$\endgroup$
  • 2
    $\begingroup$ "Multi colinearity affects the learning of Artificial Neural network. Since the information in the dependent variable is very less compared to the other variables, the neural network will take more time to converge." Do you have a source for that? $\endgroup$ – Martin Thoma Feb 26 '18 at 20:42
-2
$\begingroup$

I am playing with a Neural Network for regression tasks (i.e.: one output node) in these days. I got some data, provided by my company, that show very little heterogeneity, with groups of variables that are highly correlated. When I run a TensorFlow session, and print the loss values across the training epochs, it returns NaNs. I repeated the same model on different, larger and more variegated dataset, and it always worked good.

So, one thing to keep in mind is that excessive multicollinearity could tilt the computation of your loss function.

$\endgroup$
  • 1
    $\begingroup$ Please give more information about the network and the optimization criterion. NaNs sound very much like another problem. $\endgroup$ – Martin Thoma Dec 31 '18 at 12:05
  • $\begingroup$ Optimization algorithms return all the same results. I tried both plain GD, Adam, Adagrad, you name it. What aspect interests you? $\endgroup$ – Leevo Dec 31 '18 at 13:34
  • $\begingroup$ When I saw NaN so far, it was always a programming issue. When you don't give more details, I can't rule that out. Best would be a link to the code $\endgroup$ – Martin Thoma Dec 31 '18 at 14:56
  • $\begingroup$ I'm sure it's not a programming issue. The very same code was applied to other datasets, and it worked without problems. $\endgroup$ – Leevo Dec 31 '18 at 15:29
  • $\begingroup$ Are you talking about an MLP / CNN / LSTM network? What is your architecture? Which activation functions do you use? Which loss function? $\endgroup$ – Martin Thoma Dec 31 '18 at 15:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.