I am using chi-squared to determine feature importance as I select features to train a supervised ML model. I create a contingency table for the feature/target, and feed this contingency table into the scipy.stats.chi2_contingency module. This module returns the chi-squared value and the p-value.
I have acheived reasonable results with boolean variables, but I am suspicious of the results for categorical variables with more than 2 categories.
Specifically, I am fairly sure that one continuous feature, age, is correlated with the target, to some level of significance. From plotting histograms and KDEs, I know that the probability distribution of the feature for (target = 0) is quite different from the probability distribution for (target = 1). However, when I bin the age feature into 2-7 bins, the chi-squared test yields a p-value of ~1e-39.
Is there anything that I am missing with regards to the chi-squared test and categorical variables? Does this test only work for monotonic relationships?