I am doing an exercise of a Machine Learning System module in python that takes a dataset of cars (cylinders, year, consumption....) and asks for a model, being the variable to predict the consumption of gasoline. As it has three categorical variables, I have generated the dummies.

enter image description here

In the exercise I need to eliminate the variables with multicollinearity, so I used the method showed on my course notes:

from sklearn.linear_model import LinearRegression

def calculateVIF(data):
    features = list(data.columns)
    num_features = len(features)

    model = LinearRegression()

    result = pd.DataFrame(index = ['VIF'], columns = features)
    result = result.fillna(0)

    for ite in range(num_features):
        x_features = features[:]
        y_featue = features[ite]

        x = data[x_features]
        y = data[y_featue]

        model.fit(data[x_features], data[y_featue])

        result[y_featue] = 1/(1 - model.score(data[x_features], data[y_featue]))

    return result

Then if I launch the method it calculates a coefficient for each variable:

enter image description here

In my course notes it is said:

  • $VIF>5$ is a high value.
  • $VIF>10$ is a very high value

What should I do? I need to remove the variables that have a $VIF>10$ before executing the model?

The problem I see, for my categorical variable cylinders, is only cylinders_5 has a VIF under 10 so should I remove the others and leave cyclinders_5?


1) First, you need to do variable regression i.e for each column in your data set you do simple linear regression and calculate p-value... Thereby you get an idea of the significance of each column against the target variable.

2) plot influence plot check the cooks_d value

 import statsmodels.api as sm
  infl = model1.get_influence()
  sm_fr = infl.summary_frame()

3) You will get cooks_d value from sm_fr data frame

4)select the row point with a cooks_d value>1 and remove that row from your data frame,, now you have removed influential points. 5)Now check VIF values for new set data frame containing variables and remove the variables having vif>5 as they are insignificant ... you can also check their significance calcualting p value .

for overall procedure of building a multi linear regression model satisfying all assumotions of multilinear regression like linearity,homosedasticity,multivariate normality and no multicollineaity see the below example of prediction of profit of start- ups


| improve this answer | |
  • $\begingroup$ I will try it next week. That migth impress my teacher as the exercise only talks about removing high VIF variables. I will remove then variables with VIF>5 thanks. $\endgroup$ – user70164 Jun 29 '19 at 15:37

Never remove features from your dataset. Always try to make use of them. Try using some DR techniques like PCA to eliminate the multicollinearity between the features. Removing features means you are losing some info. unless Multicollinearity means that the correlation between them is 1 one then you can delete them safely. Using Tree-based models will capture these little differences between features.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy