I want to be able to automatically remove highly correlated features. I am performing a classification problem using a set of 20-30 features and some may be correlated.
Multiple features can be correlated at once too and I fear it may pose a problem in my Logit model significances & coefficients of the features. After removing these features I plan to then also use this reduced feature set into an Xgboost model as well.
Multicolinearity on the other hand is more troublesome to detect because it emerges when three or more variables, which are highly correlated, are included within a model. (Reference here)
Would it be correct to remove correlated variables this way?:
X=data[feature_set] vif = pd.DataFrame() vif['vif_factor'] = [variance_inflation_factor(X.values,i) for i in range(X.shape)] vif['features'] = X.columns vif.sort_values('vif_factor',axis=0,inplace=True, ascending=False) features_to_remove = vif.loc[vif['vif_factor'] > 10,'features'].values features_to_remove = list(features_to_remove) print(features_to_remove)
vif_factor | feature 21 | age 9.7 | income 7 | gender ....and so on
So in this case age would be removed as a feature from the model.