I have the following list of the names of the categorical variables in my dataset:
categorical_columns = ['MSSubClass', 'MSZoning', 'LotShape', 'LandContour', 'LotConfig', 'Neighborhood', 'Condition1', 'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 'Foundation', 'Heating', 'Electrical', 'Functional', 'GarageType', 'PavedDrive', 'Fence', 'MiscFeature', 'SaleType', 'SaleCondition', 'Street', 'CentralAir']
in this function definition, I am one-hot encoding each column with the help of a for loop:
def feature_encoding(df, categorical_list): # One Hot Encoding the columns gathered in categorical_columns for col in categorical_list: # take one-hot encoding OHE_sdf = pd.get_dummies(df[categorical_list]) # drop the old categorical column from original df df.drop(col, axis = 1, inplace = True) # attach one-hot encoded columns to original dataframe df = pd.concat([df, OHE_sdf], axis = 1, ignore_index = True)
I don't want to use all of these columns in my training so now I am at the dimensionality reduction phase of my work. I want to measure the correlation between each of these columns and my
SalePrice variable (numerical) and weed out the columns with low correlation.
I've read that Chi-square test is generally used for measuring the correlation of categorical variables but I have not seen an implementation where it was a list of categorical variables vs a continuous variable.