I have 134 categorical columns in my data. 7 of which are categorical variables [ one variable is highly unbalanced and has 34 classes while all other variables just has 3-5 classes in each variable and are almost balanced ] from data and the remaining are result of one hot encoding of a column [ all variables only have two classes ].

So, during feature selection I have performed chi-square test of dependence on my all those variables(and everything as said by this article), with hypothesis:
H0: variables are independent on each other,
H1: variables are dependent on each other.

from scipy import chi2_contingency
calculated_statistic, p_value, degrees_of_freedom, expected_values = chi2_contingency(contingency_table)

That's how I generated p-values for all variables with every remaining variable. Every variable is dependent on minimum of five other variables(p-value < 0.5).
So, my question now is do I discard all those variables.

here is the data. FYI, One-Hot encoded column is named amenities and all other categorical variables are included as they are, for testing. The variabled I'm doing all this pre-processing and feature selection to predict is continuous and named log_price in data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.