Problem Statement: Let's say I have buyer transactional data for every product, features are categorical and numeric. I want to cluster purchases that have similar attributes in terms of who's purchasing the product, their demographics etc .
I tried using hierarchical clustering and it returns certain clusters which are super similar and also clusters that do not look very similar, not sure why its even clustering those "not similar looking" orders
I am currently doing one-hot encoding for categorical variables, and minmax normalization for numeric variables.
I am using hierarchical clustering from scipy, using ward as the linkage method and using scipy.cluster.hierarchy.fcluster(Z, t, criterion='distance') to form flat clusters with t set to 1.
Questions: Is there a better way to transform categorical and numerical variables for this clustering? How can I get better clusters - currently getting orders that look dissimilar when I look at the attributes being clustered together? Should I be using a different distance function? Gower distance? Any other linkage function? Any metric I can use to measure how good my clustering is? How can I determine parameter "t" in scipy.cluster.hierarchy.fcluster Also is there a way to add weights to certain feature values to form clusters? For example, if there's a categorical feature which has a value with very high frequency in general population, I don't really want it to be given too much weightage while clustering