1
$\begingroup$

I want to be able to automatically remove highly correlated features. I am performing a classification problem using a set of 20-30 features and some may be correlated.

Multiple features can be correlated at once too and I fear it may pose a problem in my Logit model significances & coefficients of the features. After removing these features I plan to then also use this reduced feature set into an Xgboost model as well.

Multicolinearity on the other hand is more troublesome to detect because it emerges when three or more variables, which are highly correlated, are included within a model. (Reference here)

Would it be correct to remove correlated variables this way?:

X=data[feature_set]
vif = pd.DataFrame()
vif['vif_factor'] = [variance_inflation_factor(X.values,i) for i in range(X.shape[1])]
vif['features'] = X.columns

vif.sort_values('vif_factor',axis=0,inplace=True, ascending=False)

features_to_remove = vif.loc[vif['vif_factor'] > 10,'features'].values
features_to_remove = list(features_to_remove)
print(features_to_remove)

Output:

vif_factor |  feature

21         |   age
9.7        |   income
7          |   gender ....and so on 

So in this case age would be removed as a feature from the model.

$\endgroup$
  • $\begingroup$ Are you trying to do make statistical inferences about the relationship between your response and your predictors, or are you simply trying to achieve the absolute best predictions? If it is the former, by all means looking closely at multicollinearity is necessary. However, if it is the latter, why do you care so much about this? I mean, a linear logistic regression may still lead to inflated coefficients (giving high variance) but you can control this through other means like ridge regression/elastic net, rather than deleting variables outright. $\endgroup$ – aranglol May 25 '19 at 3:27
1
$\begingroup$

I might oversimplify this, but Pandas allows to drop correlated features based on a threshold. E.g. correlation >0.95

enter image description here

https://chrisalbon.com/machine_learning/feature_selection/drop_highly_correlated_features/

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.