I have 134 categorical columns in my data. 7 of which are categorical variables [ one variable is highly unbalanced and has 34 classes while all other variables just has 3-5 classes in each variable and are almost balanced ] from data and the remaining are result of one hot encoding of a column [ all variables only have two classes ].
So, during feature selection
I have performed chi-square test of dependence on my all those variables(and everything as said by this article), with hypothesis:
H0: variables are independent on each other,
H1: variables are dependent on each other.
from scipy import chi2_contingency calculated_statistic, p_value, degrees_of_freedom, expected_values = chi2_contingency(contingency_table)
That's how I generated p-values for all variables with every remaining variable.
Every variable is dependent on minimum of five other variables(p-value < 0.5).
So, my question now is do I discard all those variables.
here is the data. FYI, One-Hot encoded column is named
amenities and all other categorical variables are included as they are, for testing. The variabled I'm doing all this pre-processing and feature selection to predict is continuous and named
log_price in data.