0
$\begingroup$

I have the following list of the names of the categorical variables in my dataset:

categorical_columns = ['MSSubClass', 'MSZoning', 'LotShape', 'LandContour', 'LotConfig', 'Neighborhood', 'Condition1',
                       'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd',
                       'Foundation', 'Heating', 'Electrical', 'Functional', 'GarageType', 'PavedDrive', 'Fence',
                       'MiscFeature', 'SaleType', 'SaleCondition', 'Street', 'CentralAir']

in this function definition, I am one-hot encoding each column with the help of a for loop:

def feature_encoding(df, categorical_list):

    # One Hot Encoding the columns gathered in categorical_columns
    for col in categorical_list:

        # take one-hot encoding
        OHE_sdf = pd.get_dummies(df[categorical_list])

        # drop the old categorical column from original df
        df.drop(col, axis = 1, inplace = True)

        # attach one-hot encoded columns to original dataframe
        df = pd.concat([df, OHE_sdf], axis = 1, ignore_index = True)

I don't want to use all of these columns in my training so now I am at the dimensionality reduction phase of my work. I want to measure the correlation between each of these columns and my SalePrice variable (numerical) and weed out the columns with low correlation.

I've read that Chi-square test is generally used for measuring the correlation of categorical variables but I have not seen an implementation where it was a list of categorical variables vs a continuous variable.

$\endgroup$
2
$\begingroup$

The Chi-squared test measures the relationship between two categorical variables.

To measure the relationship between a categorical feature and a continuous feature, you can use an ANOVA test.

As an aside, you don't need to use a for loop for your encoding - get_dummies has the parameter columns which allows the user to specify which columns are encoded.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.