Problem regarding calculating correlation approach?

There are 3 columns say product_id, product_type, and price_drop. product_id is unique while a product can belong to say 100 classes that information is given by product_type. price_drop column value is 1 when a drop in the price of the product occurred else 0. What I have done is, I one-hot encoded the product_type column created the dummy variable for them. Using each dummy variable I calculated their correlation with the price_drop column. I wanted to see the correlation between each product type and a drop in price. Is this approach is correct?

• Welcome to the site Vipul. Can you explain if same product can belong to multiple product types in the question? If not then simple averaging/summary statistics by product type might suffice instead of one hot encoding. Aug 28 '20 at 12:48
• There is only one product type possible for a product. Your suggestion makes sense and I also thought about it, but I want to explore whether this approach makes sense and if not what is wrong with this approach? Aug 28 '20 at 16:32

Most probably, you are using Pearson's correlation method. This method is used for two Continuous features.
Here, both the price_drop and the OHE features are Binary Categorical features.

So, you can use these methods -
Phi - Phi is a measure of the degree of association between two binary variables (two categorical variables, each of which can have only one of two values)
Crammer's V - Cramer’s V is an extension of phi for tables larger than 2×2.
Both are extensions of the Chi-square test of Independence.

Since both the Features have 2 values, both of the above methods will output the same result.

s1 = dataset['Status']
s2 = dataset[product_type_OHE_01]

import pandas as pd
from scipy.stats import chi2_contingency
n = len(s1)
r,c = s1.nunique(), s2.nunique()
matrix = pd.crosstab(s1,s2).values
chi_sq = chi2_contingency(matrix)

phi = np.sqrt(chi_sq/n)
cramm_V = np.sqrt(chi_sq/(n*min(r-1,c-1)))

print(phi, cramm_V)
• Will it make sense, if I Instead of doing one-hot encode product_type. I found out the gamma coefficient (Kruskal gamma) between product_type and price_drop instead of the approach suggested by you. Aug 28 '20 at 16:25
• 1. This method is for ordinal data 2. You "wanted to see the correlation between each product type and the drop in price". 3. Even if dont want for each product type, you should use Crammers V for type vs drop Aug 28 '20 at 16:47
• This is the correct method since both features are binary. Aug 29 '20 at 9:06