I am working with PUBG data and developing a linear regression model for the same ! Now there were three features in my original dataset, ridedistance, swimdistance, walkdistance. I combined the three with a new feature : distance covered which is the sum of the above mentioned three features. When putting it in a linearregression model, when I use the three features and the fourth one as well, I get a better score as compared to using the three featues only or using only the fourth feature. I have read that correlation between features when developing a model should not be there. But when all features (4 of them) having correlation are used to develop a model, the model has a better square (R-square). Why is this happening ?
$\begingroup$
$\endgroup$
6
asked Jan 27, 2019 at 12:30
-
$\begingroup$ It would be better if you specify the model...is it a NN linear regression model? or simply a ML regression model? Also what are you trying to predict? $\endgroup$– DuttaACommented Jan 27, 2019 at 16:49
-
$\begingroup$ @DuttaA It is a linear regression model from scikit-learn. There are 29 total columns (all continuous) and I am trying to predict a float number from 0 to 1 which can have any value upto 2 places of decimal. $\endgroup$– Rishabh SharmaCommented Jan 27, 2019 at 16:50
-
$\begingroup$ I don't know what sickit-learn is using, so you need to specify what exactly are you using...whether it is a NN based model or not.. $\endgroup$– DuttaACommented Jan 27, 2019 at 16:52
-
$\begingroup$ It;s not NN based model. It's a model based on gradient descent and regression much like y_predict = m1x1 + m2x2 + ... + c $\endgroup$– Rishabh SharmaCommented Jan 27, 2019 at 16:58
-
$\begingroup$ I am guessing it is an overfit model. Also, I doubt if there will be collinearity among the first 3 - the collinearity would exist between the each of the first 3 with the fourth variable. $\endgroup$– VivekCommented Feb 22, 2020 at 13:29
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
It seems that you are dealing with the problem of Multicollinearity. Multicollinearity happens when your predictors are correlated with other predictors in the model.
Moderate multicollinearity may not be problematic. However, severe multicollinearity is a problem because it can increase the variance of the coefficient estimates and make the estimates very sensitive to minor changes in the model. The result is that the coefficient estimates are unstable and difficult to interpret.
You can use adjusted R-squared to see if the new added variable is actually helping your model to better explain the variance.
