0
$\begingroup$

I am using chi-squared to determine feature importance as I select features to train a supervised ML model. I create a contingency table for the feature/target, and feed this contingency table into the scipy.stats.chi2_contingency module. This module returns the chi-squared value and the p-value.

I have acheived reasonable results with boolean variables, but I am suspicious of the results for categorical variables with more than 2 categories.

Specifically, I am fairly sure that one continuous feature, age, is correlated with the target, to some level of significance. From plotting histograms and KDEs, I know that the probability distribution of the feature for (target = 0) is quite different from the probability distribution for (target = 1). However, when I bin the age feature into 2-7 bins, the chi-squared test yields a p-value of ~1e-39.

Is there anything that I am missing with regards to the chi-squared test and categorical variables? Does this test only work for monotonic relationships?

$\endgroup$
  • $\begingroup$ your p-value is good according to chi square test (null hypothesis is independence) $\endgroup$ – Elliot Jun 4 '18 at 14:16
1
$\begingroup$

It sounds like the chi-squared test confirms your suspicions about age being correlated with the response.

As far as I am aware, the null-hypothesis for the chi-squared test is "there is no relationship" between the two variables. The test statistic is calculated based on the assumption that all observations are evenly distributed amongst the cells in the contingency table, so the test should work for most kinds of relationships.

A word of warning - the test is sensitive to data imbalances.

$\endgroup$
  • $\begingroup$ This makes sense recalling my experience with the chi-squared test. It seems that this test is most often used to reject the null-hypothesis that the variables are independent. I intend to use the complement -i.e., using the p-value to confirm the H1 hypothesis that the variables are dependent. For example, if the p-value is 0.05, I would interpret this to mean that there the variables are dependent up to the 95% confidence level. Are there any pitfalls to interpreting chi-squared in this way? $\endgroup$ – fermi Jun 6 '18 at 6:06
  • $\begingroup$ I think that approach is fine. You could also repeatedly randomly sub-sample your data, perform the chi-square test and calculate the p-value. This way you get a distribution of p-values and can be more certain of the results. On another note, the chi-square test doesn't highlight the strength of the relationship - if you're interested in that, then you should perform a Cramer's V test. $\endgroup$ – bradS Jun 6 '18 at 8:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.