2
$\begingroup$

I am doing feature selection of features which are of binary nature i.e. each feature represents presence or absence of a substructure in a molecule. And I have a target variable of two classes. My first step was to check if my feature is associated with the target variable so I was using chi-square test for that.And later I check if the selected features have any correlation and remove that. So each feature is checked for association with target variable using chi square test. I am wondering if any correction methods are needed here in my case like bonferroni or false discovery rate. If yes what are the assumptions I should keep in mind. Really appreciate the help.

$\endgroup$
8
  • 1
    $\begingroup$ You're removing features that are correlated with the outcome? $\endgroup$
    – Dave
    Nov 30, 2023 at 18:39
  • $\begingroup$ I'm trying to check if features are associated with the target variable. For that, I'm using chi-square test hypothesis testing. $\endgroup$
    – As13
    Dec 1, 2023 at 4:30
  • $\begingroup$ Could you please clarify what your criteria are for eliminating features? $\endgroup$
    – Dave
    Dec 1, 2023 at 4:36
  • $\begingroup$ So I have made two hypothesis; Null Hypothesis 𝐻0 : Feature & Target Variable are not dependent ● Alternate Hypothesis 𝐻1 : Feature & Target Variable are dependent So for the feature with a p-value less than chosen significance level α=0.05, the null hypothesis is rejected. And we say that the feature and target variable are associated. This is done for all features in a for loop. $\endgroup$
    – As13
    Dec 1, 2023 at 4:42
  • $\begingroup$ So you keep the variable when the p-value is what? $\endgroup$
    – Dave
    Dec 1, 2023 at 4:50

1 Answer 1

2
$\begingroup$

In a basic statistical model like a logistic regression on the features, this kind of univariate feature selection is problematic. Briefly, a candidate feature can be fairly unrelated to the outcome on its own but become quite important once another feature is considered. If you do univariate screening, you miss out on that kind of important feature.

If you use a more sophisticated machine learning model like a random forest or a neural network, even a logistic regression with feature interactions, then univariate screening is even more problematic. Consider the image below (taken from another post of mine).

red/blue X

Jointly, the two features almost perfectly distinguish between red and blue. However, neither feature, alone, is at all correlated with the color. If you have interaction terms in a logistic regression or run a model like a random forest or neural network that will interact features as it sees fit, then you miss out on these being important features, even if their importance only happens in the interaction.

You can explicitly test for a correlation between the interaction and the outcome, which would reveal the interaction in this case to be extremely important, but then you're doing univariate screening (the interaction feature is, after all, a candidate feature), which is problematic as described above.

(I consider a chi-squared test to be a correlation-style measure, due to the relationship between chi-squared testing and a score test in a logistic regression, so I am content to use a bit of slang when I write "correlation".)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.