I have a task which is customer segmentation with 120k users and a record of their purchases which is +3 million records of data, the approach I want to use is to use clustering algorithms like kmeans or pca and then analyze the clusters .

I have 2 questions here:

  1. what are the ways to analyze and interpret clusters ?

  2. most of clustering algorithms that i want to fit raise the MemoryError exception, due to 3M record of data, what should i do?

  • $\begingroup$ Can you describe what information a purchase record contains? $\endgroup$
    – noe
    Feb 22, 2023 at 12:46

1 Answer 1


1- Establish general rules and patterns with a random sample of ~2% of total rows. Even if the data is significant, you have always to get "the big picture" using a random and representative sample.

2- Apply algorithms such as PCA or KMeans. But you should also use dimensional reduction algorithms such as T-SNE or UMAP. Business intelligence tools like Power BI or Qlik are interesting because they are built to visualize significant data easily. In addition to that, the algorithms' results can be plugged into them.

3- Once you identify data groups, you can increase the quantity. But always keep in mind whether it is useful or not. Do you want to extract general information from each group, or do you want to look precisely at it?

4- Eventually, once you master algorithms in small datasets, you can use the algorithms in all data or big data groups without making too many errors. One of the main issues in dealing with large datasets is the long processing time. If you try the algorithms on the whole dataset from the start, you will probably get poor results because the parameters are wrong. But if you start with simpler, fewer but representative random data, you will finally have a coherent and progressive approach to reach the 3M rows much faster with fewer errors.

  • $\begingroup$ 3. I want to look at every clustering to find relationship between feature to extract some rules to segment my customers. and also my data has date time element in it, which is the time that a specific customer buy a specific product. in this case how should i sample my data ? $\endgroup$ Feb 23, 2023 at 9:44
  • $\begingroup$ Start simple: select a short but significant period to extract meaningful results with your algorithms. You will have to study different approaches (by groups of customers, location, period, etc.) to find the better ones. $\endgroup$ Feb 23, 2023 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.