0
$\begingroup$

I have 134 categorical columns in my data. 7 of which are categorical variables [ one variable is highly unbalanced and has 34 classes while all other variables just has 3-5 classes in each variable and are almost balanced ] from data and the remaining are result of one hot encoding of a column [ all variables only have two classes ].

So, during feature selection I have performed chi-square test of dependence on my all those variables(and everything as said by this article), with hypothesis:
H0: variables are independent on each other,
H1: variables are dependent on each other.

from scipy import chi2_contingency
calculated_statistic, p_value, degrees_of_freedom, expected_values = chi2_contingency(contingency_table)

That's how I generated p-values for all variables with every remaining variable. Every variable is dependent on minimum of five other variables(p-value < 0.5).
So, my question now is do I discard all those variables.

here is the data. FYI, One-Hot encoded column is named amenities and all other categorical variables are included as they are, for testing. The variabled I'm doing all this pre-processing and feature selection to predict is continuous and named log_price in data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.