2
$\begingroup$

I want to be able to automatically remove highly correlated features. I am performing a classification problem using a set of 20-30 features and some may be correlated.

Multiple features can be correlated at once too and I fear it may pose a problem in my Logit model significances & coefficients of the features. After removing these features I plan to then also use this reduced feature set into an Xgboost model as well.

Multicolinearity on the other hand is more troublesome to detect because it emerges when three or more variables, which are highly correlated, are included within a model. (Reference here)

Would it be correct to remove correlated variables this way?:

X=data[feature_set]
vif = pd.DataFrame()
vif['vif_factor'] = [variance_inflation_factor(X.values,i) for i in range(X.shape[1])]
vif['features'] = X.columns

vif.sort_values('vif_factor',axis=0,inplace=True, ascending=False)

features_to_remove = vif.loc[vif['vif_factor'] > 10,'features'].values
features_to_remove = list(features_to_remove)
print(features_to_remove)

Output: 

vif_factor |  feature

21         |   age
9.7        |   income
7          |   gender ....and so on

So in this case age would be removed as a feature from the model.

Improve this question
$\endgroup$
1
  • $\begingroup$ Are you trying to do make statistical inferences about the relationship between your response and your predictors, or are you simply trying to achieve the absolute best predictions? If it is the former, by all means looking closely at multicollinearity is necessary. However, if it is the latter, why do you care so much about this? I mean, a linear logistic regression may still lead to inflated coefficients (giving high variance) but you can control this through other means like ridge regression/elastic net, rather than deleting variables outright. $\endgroup$
    – aranglol
    May 25, 2019 at 3:27

1 Answer 1

Reset to default
2
$\begingroup$

I might oversimplify this, but Pandas allows to drop correlated features based on a threshold. E.g. correlation >0.95

enter image description here

https://chrisalbon.com/machine_learning/feature_selection/drop_highly_correlated_features/

Improve this answer
$\endgroup$
1
  • $\begingroup$ I think this solution may excessively remove features. If x and y are pairwise correlated and y and z are pairwise correlated, this would first check correlations with x, so we'd remove y. But then we'd still be checking correlations with y despite it no longer being necessary, and we'd end up removing z unnecessarily. $\endgroup$
    – David
    Sep 4, 2022 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.