0
$\begingroup$

I have data of sales, however it is in the millions, about 500M rows. I aggregate this data by factors such as location, shoptype, country_of_shop, cardtype, and then the aggregated statistic is:

  • number of transactions
  • sum of amount in dollars

My questions is can i cluster this say using kprototypes or something similar? And can i reduce the dimensionality using Factor analysis?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

There are no clear restrictions on can you cluster the data or not based on the data itself (in most of the cases), but you might face restrictions based on computational speed. The question is what you want to achieve in the end, but you can for sure cluster the data or reduce dimensions. You can also take a subset of your data, cluster it and then train ML model, something like catboost or lightgbm to predict clusters, instead of doing clustering on whole dataset. In my experience it will be faster to use model for cluster predictions.

For dimension reduction, you can go for PCA or TruncatedSVD, here are some useful links:

https://scikit-learn.org/stable/modules/unsupervised_reduction.html https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html

But remember, that this techniques will reduce the number of features, not the rows, so you can still face computational issues.

If you want a way to map your high-dimensional data (around ~50 dimensions or less usually), you can use t-SNE https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html

Improve this answer
$\endgroup$
3
  • $\begingroup$ using PCA or t-SNE however will lose meaning on categorical data no? $\endgroup$
    – Maths12
    Apr 7, 2021 at 19:31
  • $\begingroup$ you will need to encode categorical data, since there is no native support for categorical columns for either pca or t-sne. $\endgroup$
    – Yaroslaw Homenko
    Apr 7, 2021 at 19:42
  • $\begingroup$ but encoding categorical data e.g. one hot encoding it makes PCA or t-SNE less affect since e.g. for PCA It tries to minimize variance (=squared deviations). The concept of squared deviations breaks down when you have binary variables $\endgroup$
    – Maths12
    Apr 7, 2021 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.