I am trying to do a customer segmentation on my transactional data and I am struggling a little bit on the best approach. Since it is an unsupervised model I can throw it to any algorithm and get some clusters but I am more interested in the best approach to do it.
My data has basically 3 different products. Each product has a dozens of features. The issue is that one product is purchased by 95% of the customer while the other 2 are only purchased by 10 to 20% of the customers ( each customer can purchase any of the 3 products multiple time). It means that my dataset has a lot of zeros for product 2 and 3 (which is also informative).
I have done segmentation on each product individually and now I am trying to do a clustering based on all the data . I basically merged 3 datasets by customer to get one file.
Note that the data file include 1 to 2 millions of rows.
I have looked at the distribution of every variables and while some have a nicely normal distribution, many other have peak at 0 and sometimes 1 ( I have several percentage variables). Some examples below: enter image description here
After some data cleaning I used K-mean in pyspark to get some clusters. I can see some interesting segment based on the distribution of some of the features in the cluster. However I am not sure if using euclidian distance is the best option in my case. I have been thinking of trying other algorithm such as k- medoid but it also come with challenges as I need to transform to panda data frame and then run it in python which might fail with so much data.
What would be the best way to handle this scenario when you have features for 3 different products but 2 of them are not purchased as much as the first one?All the features are numerical in my file but I also thought about adding binary variables to tell if customer purchased the product or not and then use them as features.