I am doing a feature selection for a data science project with one of those feature being a high cardinality categorical variable (for context, it’s nationality). I know chi-square test could handle multiclass feature like mine but I need to do one-hot encode (dividing a multiclass variable into multiple binary variable based on its values) to be able to input it into my machine learning algorithm (spark mllib). My question is does doing one-hot encode effects the result of a chi-square test? I think it will because the values of one are based on all the other values is it not? Sorry for the harsh english and thank you in advance.
1
-
$\begingroup$ Are you using the one hot encoded values as input to the chi-square test? $\endgroup$– Ralph WintersCommented May 16, 2022 at 13:15
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I suppose you are talking about Chi-square independence test. The answer is no, simply because in order to perform the test you need to transform your single multiclass variable into a one hot encoding vector.
Also, the assumptions from chi square test includes the facts that a single observation must be a distinct choice between classes. For example an observation should be either masculine or feminine for a binary variable about gender, and it should not be possible that one observation to be both, or neither. Otherwise the math behind chi square independence test will not work and the results will be random.
As a final note, be aware if the counters. A rule of thumb is to have at least 5 observations in each cell. If this is not the case, one way to alleviate the issue is to group some categories, but of course, you will receive an answer to a different question, even if that question is very close to your original question.
