4
$\begingroup$

Im working on selecting most effective features from a dataset with over that 2000 features. Im using different algorithms for that (selectKBest with chi-square, Extra Trees, Correlation etc.) But when I look the features ranking I saw that selectKBest with chi-square are generating excatly same results as Correlation. Is it possible or am I doing something wrong?

My all features consist of 64bit float continuous numbers, between [-8,11] and my target column is binary which can be only 0 or 1.

Updated on 05.09.19: I am STILL searching how can it be possible? I mean I can guess that both methods based on same formula and developed by same person but I need a proof for understand that clearly.

Correlation Function:

cor = data.corr()
# Class is my target column
cor_target = abs(cor["Class"])
# Want to get correlation values for every feature without target column
relevant_features = cor_target[cor_target > 0].drop(labels=["Class"])
#Top 1000 features
relevant_features = pd.Series(relevant_features, index=data.columns).nlargest(1000).index.values

SelectKBest function:

bestfeatures = SelectKBest(score_func=chi2, k="all")
fit = bestfeatures.fit(dataValues, dataTargetEncoded)
feat_importances_chi = pd.Series(fit.scores_, index=dataValues.columns).nlargest(1000).index.values

And the result relevant_features and feat_importances_chi have excatly same results.

Improve this question
$\endgroup$
4
  • $\begingroup$ What kind of features do you have? Numeric ? Categorical? Binary? $\endgroup$
    – astel
    Aug 28, 2019 at 14:06
  • $\begingroup$ Thank you for answer. I updated my question. $\endgroup$
    – justRandomLearner
    Aug 29, 2019 at 13:35
  • 2
    $\begingroup$ I guess this first thing I will suggest is that the chi squared statistic is intended for categorical variables not for continuous variables. There is likely some binning done internally by your function but I can't find any documentation on how. $\endgroup$
    – astel
    Aug 29, 2019 at 14:51
  • $\begingroup$ Yes, in general there are deficiencies in the documentation but I asked a question about it maybe it helps you too: stackoverflow.com/questions/57273694/… $\endgroup$
    – justRandomLearner
    Sep 2, 2019 at 11:49

1 Answer 1

Reset to default
0
$\begingroup$

Likely what you are seeing is that those features that are highly correlated with each other are overweighting the results, so that if one of the features describes the target well, then the other highly correlated (or uncorrelated) features also match the target just as well.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I understand. I was expecting nearly the same results, but I was quite skeptical that the ranking of the 2000 attributes was exactly the same. $\endgroup$
    – justRandomLearner
    Aug 13, 2019 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.