# Automatically Remove highly correlated features using Variance Inflation Factor?

I want to be able to automatically remove highly correlated features. I am performing a classification problem using a set of 20-30 features and some may be correlated.

Multiple features can be correlated at once too and I fear it may pose a problem in my Logit model significances & coefficients of the features. After removing these features I plan to then also use this reduced feature set into an Xgboost model as well.

Multicolinearity on the other hand is more troublesome to detect because it emerges when three or more variables, which are highly correlated, are included within a model. (Reference here)

Would it be correct to remove correlated variables this way?:

X=data[feature_set]
vif = pd.DataFrame()
vif['vif_factor'] = [variance_inflation_factor(X.values,i) for i in range(X.shape[1])]
vif['features'] = X.columns

vif.sort_values('vif_factor',axis=0,inplace=True, ascending=False)

features_to_remove = vif.loc[vif['vif_factor'] > 10,'features'].values
features_to_remove = list(features_to_remove)
print(features_to_remove)


Output:

vif_factor |  feature

21         |   age
9.7        |   income
7          |   gender ....and so on


So in this case age would be removed as a feature from the model.

• Are you trying to do make statistical inferences about the relationship between your response and your predictors, or are you simply trying to achieve the absolute best predictions? If it is the former, by all means looking closely at multicollinearity is necessary. However, if it is the latter, why do you care so much about this? I mean, a linear logistic regression may still lead to inflated coefficients (giving high variance) but you can control this through other means like ridge regression/elastic net, rather than deleting variables outright. May 25, 2019 at 3:27