# Multicollinearity(Variance Inflation Factor). Variables to remove before doing a model

I am doing an exercise of a Machine Learning System module in python that takes a dataset of cars (cylinders, year, consumption....) and asks for a model, being the variable to predict the consumption of gasoline. As it has three categorical variables, I have generated the dummies. In the exercise I need to eliminate the variables with multicollinearity, so I used the method showed on my course notes:

from sklearn.linear_model import LinearRegression

def calculateVIF(data):
features = list(data.columns)
num_features = len(features)

model = LinearRegression()

result = pd.DataFrame(index = ['VIF'], columns = features)
result = result.fillna(0)

for ite in range(num_features):
x_features = features[:]
y_featue = features[ite]
x_features.remove(y_featue)

x = data[x_features]
y = data[y_featue]

model.fit(data[x_features], data[y_featue])

result[y_featue] = 1/(1 - model.score(data[x_features], data[y_featue]))

return result


Then if I launch the method it calculates a coefficient for each variable: In my course notes it is said:

• $$VIF>5$$ is a high value.
• $$VIF>10$$ is a very high value

What should I do? I need to remove the variables that have a $$VIF>10$$ before executing the model?

The problem I see, for my categorical variable cylinders, is only cylinders_5 has a VIF under 10 so should I remove the others and leave cyclinders_5?

• You have various option of checking the correlation of input and output variable. you can go with correlation matrix, VIF, Heatmap. if You have to deal multicollinearity then you have two option 1.Combian highly corelated feature 2.Penilize or remove highly corelated features. Oct 14, 2022 at 3:32

1) First, you need to do variable regression i.e for each column in your data set you do simple linear regression and calculate p-value... Thereby you get an idea of the significance of each column against the target variable.

2) plot influence plot check the cooks_d value

 import statsmodels.api as sm
infl = model1.get_influence()
sm_fr = infl.summary_frame()


3) You will get cooks_d value from sm_fr data frame

4)select the row point with a cooks_d value>1 and remove that row from your data frame,, now you have removed influential points. 5)Now check VIF values for new set data frame containing variables and remove the variables having vif>5 as they are insignificant ... you can also check their significance calcualting p value .

for overall procedure of building a multi linear regression model satisfying all assumotions of multilinear regression like linearity,homosedasticity,multivariate normality and no multicollineaity see the below example of prediction of profit of start- ups

https://github.com/tharun435/Data-Science-/blob/master/startups.ipynb

• I will try it next week. That migth impress my teacher as the exercise only talks about removing high VIF variables. I will remove then variables with VIF>5 thanks.
– user70164
Jun 29, 2019 at 15:37

Here is a code I have written to handle Multicollinearity in a dataset. This code snippet is able to handle the following listed items:

• Multicollinearity using Variable Inflation Factor (VIF), set to a default threshold of 5.0
• You just need to pass the dataframe, containing just those columns on which you want to test multicollinearity.
• This function will drop those columns which contains just 1 value. For a bit more further details on this point, please have a look my answer on How to run a multicollinearity test on a pandas dataframe?.
• The calculation of VIF is parallelized over multiple cores.
    from joblib import Parallel, delayed
from statsmodels.stats.outliers_influence import variance_inflation_factor

def removeMultiColl(data, vif_threshold = 5.0):
for i in data.columns:
if data[i].nunique() == 1:
print(f"Dropping {i} due to just 1 unique value")
data.drop(columns = i, inplace = True)
drop = True
col_list = list(data.columns)
while drop == True:
drop = False
vif_list = Parallel(n_jobs = -1, verbose = 5)(delayed(variance_inflation_factor)(data[col_list].values, i) for i in range(data[col_list].shape))
max_index = vif_list.index(max(vif_list))
if vif_list[max_index] > vif_threshold:
print(f"Dropping column : {col_list[max_index]} at index - {max_index}")
del col_list[max_index]
drop = True
print("Remaining columns :\n", list(data[col_list].columns))
return data[col_list]


Good luck !

Never remove features from your dataset. Always try to make use of them. Try using some DR techniques like PCA to eliminate the multicollinearity between the features. Removing features means you are losing some info. unless Multicollinearity means that the correlation between them is 1 one then you can delete them safely. Using Tree-based models will capture these little differences between features.