There are 3 columns say product_id, product_type, and price_drop. product_id is unique while a product can belong to say 100 classes that information is given by product_type. price_drop column value is 1 when a drop in the price of the product occurred else 0. What I have done is, I one-hot encoded the product_type column created the dummy variable for them. Using each dummy variable I calculated their correlation with the price_drop column. I wanted to see the correlation between each product type and a drop in price. Is this approach is correct?

  • $\begingroup$ Welcome to the site Vipul. Can you explain if same product can belong to multiple product types in the question? If not then simple averaging/summary statistics by product type might suffice instead of one hot encoding. $\endgroup$ – hssay Aug 28 at 12:48
  • $\begingroup$ There is only one product type possible for a product. Your suggestion makes sense and I also thought about it, but I want to explore whether this approach makes sense and if not what is wrong with this approach? $\endgroup$ – Vipul Jain Aug 28 at 16:32

Most probably, you are using Pearson's correlation method. This method is used for two Continuous features.
Here, both the price_drop and the OHE features are Binary Categorical features.

So, you can use these methods -
Phi - Phi is a measure of the degree of association between two binary variables (two categorical variables, each of which can have only one of two values)
Crammer's V - Cramer’s V is an extension of phi for tables larger than 2×2.
Both are extensions of the Chi-square test of Independence.

Since both the Features have 2 values, both of the above methods will output the same result.

# dataset is your DataFrame
s1 = dataset['Status']
s2 = dataset[product_type_OHE_01]

import pandas as pd
from scipy.stats import chi2_contingency
n = len(s1)
r,c = s1.nunique(), s2.nunique()
matrix = pd.crosstab(s1,s2).values
chi_sq = chi2_contingency(matrix)

phi = np.sqrt(chi_sq[0]/n)
cramm_V = np.sqrt(chi_sq[0]/(n*min(r-1,c-1)))

print(phi, cramm_V)
| improve this answer | |
  • $\begingroup$ Will it make sense, if I Instead of doing one-hot encode product_type. I found out the gamma coefficient (Kruskal gamma) between product_type and price_drop instead of the approach suggested by you. $\endgroup$ – Vipul Jain Aug 28 at 16:25
  • 1
    $\begingroup$ 1. This method is for ordinal data 2. You "wanted to see the correlation between each product type and the drop in price". 3. Even if dont want for each product type, you should use Crammers V for type vs drop $\endgroup$ – 10xAI Aug 28 at 16:47
  • $\begingroup$ This is the correct method since both features are binary. $\endgroup$ – hssay Aug 29 at 9:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.