customer segmentation with unbalanced data

Question

I am trying to do a customer segmentation on my transactional data and I am struggling a little bit on the best approach. Since it is an unsupervised model I can throw it to any algorithm and get some clusters but I am more interested in the best approach to do it.

My data has basically 3 different products. Each product has a dozens of features. The issue is that one product is purchased by 95% of the customer while the other 2 are only purchased by 10 to 20% of the customers ( each customer can purchase any of the 3 products multiple time). It means that my dataset has a lot of zeros for product 2 and 3 (which is also informative).

I have done segmentation on each product individually and now I am trying to do a clustering based on all the data . I basically merged 3 datasets by customer to get one file.

Note that the data file include 1 to 2 millions of rows.

I have looked at the distribution of every variables and while some have a nicely normal distribution, many other have peak at 0 and sometimes 1 ( I have several percentage variables). Some examples below: enter image description here

After some data cleaning I used K-mean in pyspark to get some clusters. I can see some interesting segment based on the distribution of some of the features in the cluster. However I am not sure if using euclidian distance is the best option in my case. I have been thinking of trying other algorithm such as k- medoid but it also come with challenges as I need to transform to panda data frame and then run it in python which might fail with so much data.

What would be the best way to handle this scenario when you have features for 3 different products but 2 of them are not purchased as much as the first one?All the features are numerical in my file but I also thought about adding binary variables to tell if customer purchased the product or not and then use them as features.

Answer 1

Split your data into zero and non-zero valued to get like 8 clusters.

That is by far the best clustering you can find in your data because of the data. I doubt you can find much more than that. Have you tested if any of the "interesting" patterns you found with k-means are better than these?

Answer 2

At first, I would consider zeros and non-zeros as "nonpayers" and "payers" respectively. You can split nonpayers and segments them by features not related to transactions - activity or user age (days from first "session"/install).

Also, segmentation doesn not necessary mean usage of ML models. If it's suitable for you I'd look for RFM segmentation as well (as a very simple approach). In RFM you have 3 variables for each customer:

1) Recency - the time from the last transaction
2) Frequency - a number of transactions
3) Monetary - total amount of money spent

Basically you can do this for any of your product separately as well. The main advantages of this approach are interpretability and simplicity (low computation resources as well). There're some good tutorials on RFM you can find just searching for "RFM segmentation in python". Here are some random ones (except first):

• Lifetimes library - I really like this one, but not sure how to use it with spark
• Random tutorial #1
• Random tutorial #2

Answer 3

I just created a small optimization based customer segmentation model that seems like it would address your use case. It essentially relies on probabilities from a binary classification model along with the historical binary outcome of a purchase / action.

Feel free to check out the links:

https://github.com/astronights/smart-segment

https://pypi.org/project/smart-segment/