I am at the dimensionality reduction phase of my model. I have a list of categorical columns and I want to find the correlation between each column and my continuous SalePrice column. Below is the list of column names:

categorical_columns = ['MSSubClass', 'MSZoning', 'LotShape', 'LandContour', 'LotConfig', 'Neighborhood', 'Condition1',
                       'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd',
                       'Foundation', 'Heating', 'Electrical', 'Functional', 'GarageType', 'PavedDrive', 'Fence',
                       'MiscFeature', 'SaleType', 'SaleCondition', 'Street', 'CentralAir']

Because its categorical vs continuous, I've read that ANOVA is the best way to go but I have never used it before and couldn't find a concise implementation of it in Python. I want to loop through and output the correlation between each element in the list and the SalePrice column.


I am not sure ANOVA is the best and easiest way to find correlation between these categorical features and your target. You may see this great post where they propose many other methods along with ANOVA. If you persist to use ANOVA test or Kruskal-Wallis H Test, you need to know how it works to give you that notion of correlation (variation of variance among groups of categoricals). It is nicely explained in that post:

ANOVA estimates the variance of the continuous variable that can be explained through the categorical variable. One need to group the continuous variable using the categorical variable, measure the variance in each group and comparing it to the overall variance of the continuous variable. If the variance after grouping falls down significantly, it means that the categorical variable can explain most of the variance of the continuous variable and so the two variables likely have a strong association. If the variables have no correlation, then the variance in the groups is expected to be similar to the original variance.

Once you understand how it works, implementing it and automating it is not difficult. In fact scipy and statsmodels have ANOVA. Check this post out, where they demonstrate in details how to perform ANOVA test on an actual dataset and estimate the correlation between categorical variable and continuous target. It is just a matter of putting these pieces together and change a bit to make it work for your own dataframe.

  • $\begingroup$ What would you suggest for this if not ANOVA? $\endgroup$ Jul 15 '19 at 8:20
  • $\begingroup$ Did you see that post? There were some suggestions. I personally do not have good experience for correlation of cat. and num., usually I end up training a model e.g. a GBT and look at dependency plots like SHAP values to infer alike-correlation conclusions. $\endgroup$ Jul 15 '19 at 20:16
  • $\begingroup$ I did but it just gave a list of suggestions. I have decided to go with a 1-way ANOVA using Python but now I'm trying to figure out how to do that right after one-hot encoding my categorical variables. $\endgroup$ Jul 16 '19 at 4:03
  • $\begingroup$ OK, this brings me to ask you why you do one-hot encoding, your ML model or..?! One-hot encoding is one of my no-go methods, of course depending on what model you wanna pick. Check this recent great post benchmarking alternative categorical encoding methods: towardsdatascience.com/…. $\endgroup$ Jul 16 '19 at 5:25
  • $\begingroup$ Why are you against OHE? $\endgroup$ Jul 16 '19 at 8:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.