I am working with PUBG data and developing a linear regression model for the same ! Now there were three features in my original dataset, ridedistance, swimdistance, walkdistance. I combined the three with a new feature : distance covered which is the sum of the above mentioned three features. When putting it in a linearregression model, when I use the three features and the fourth one as well, I get a better score as compared to using the three featues only or using only the fourth feature. I have read that correlation between features when developing a model should not be there. But when all features (4 of them) having correlation are used to develop a model, the model has a better square (R-square). Why is this happening ?

  • $\begingroup$ It would be better if you specify the model...is it a NN linear regression model? or simply a ML regression model? Also what are you trying to predict? $\endgroup$ – DuttaA Jan 27 '19 at 16:49
  • $\begingroup$ @DuttaA It is a linear regression model from scikit-learn. There are 29 total columns (all continuous) and I am trying to predict a float number from 0 to 1 which can have any value upto 2 places of decimal. $\endgroup$ – Rishabh Sharma Jan 27 '19 at 16:50
  • $\begingroup$ I don't know what sickit-learn is using, so you need to specify what exactly are you using...whether it is a NN based model or not.. $\endgroup$ – DuttaA Jan 27 '19 at 16:52
  • $\begingroup$ It;s not NN based model. It's a model based on gradient descent and regression much like y_predict = m1x1 + m2x2 + ... + c $\endgroup$ – Rishabh Sharma Jan 27 '19 at 16:58

It seems that you are dealing with the problem of Multicollinearity. Multicollinearity happens when your predictors are correlated with other predictors in the model.

Moderate multicollinearity may not be problematic. However, severe multicollinearity is a problem because it can increase the variance of the coefficient estimates and make the estimates very sensitive to minor changes in the model. The result is that the coefficient estimates are unstable and difficult to interpret.

You can use adjusted R-squared to see if the new added variable is actually helping your model to better explain the variance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.