Im working on selecting most effective features from a dataset with over that 2000 features. Im using different algorithms for that (selectKBest with chi-square, Extra Trees, Correlation etc.) But when I look the features ranking I saw that selectKBest with chi-square are generating excatly same results as Correlation. Is it possible or am I doing something wrong?

Correlation Function:

cor = data.corr()
# Class is my target column
cor_target = abs(cor["Class"])
# Want to get correlation values for every feature without target column
relevant_features = cor_target[cor_target > 0].drop(labels=["Class"])
#Top 1000 features
relevant_features = pd.Series(relevant_features, index=data.columns).nlargest(1000).index.values

SelectKBest function:

bestfeatures = SelectKBest(score_func=chi2, k="all")
fit = bestfeatures.fit(dataValues, dataTargetEncoded)
feat_importances_chi = pd.Series(fit.scores_, index=dataValues.columns).nlargest(1000).index.values

And the result relevant_features and feat_importances_chi have excatly same results.


Likely what you are seeing is that those features that are highly correlated with each other are overweighting the results, so that if one of the features describes the target well, then the other highly correlated (or uncorrelated) features also match the target just as well.

  • $\begingroup$ I understand. I was expecting nearly the same results, but I was quite skeptical that the ranking of the 2000 attributes was exactly the same. $\endgroup$ – justRandomLearner Aug 13 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.