0
$\begingroup$

we are currently performing a K-MEANS under scikit-learn on a data set containing 236027 observations with 6 variables in double format (64 bits).

According to our calculations, the complexity of the algorithm is O(n * k * v * i), with n the number of observations, k the number of clusters, v the number of variables and i the number of iterations which is 300 maximum.

Let (236027 * 3 * 6 * 300) * 64 (size of the double in bits): we have 81570931200 bits which is ~10.1 GB of memory.

However, by running the algorithm on a 600 GB VM, it crashed due to lack of memory.

Are we wrong in our calculation of the space required for KMEANS?

Is KMEANS suitable for this volume?

What solutions are available to us?

Thanks you

$\endgroup$
1
$\begingroup$

You should consider hierarchical K-Means Clustering.

Link to show an idea:

https://www.datanovia.com/en/lessons/hierarchical-k-means-clustering-optimize-clusters/

| improve this answer | |
$\endgroup$
0
$\begingroup$

I'm not sure about size of double in python is 64 bits.

You can use mini batch configuration of kmeans: https://scikit-learn.org/stable/modules/clustering.html#mini-batch-kmeans

also maybe DBSCAN will be suitable for you

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for advice, but it seems that KMeans can handle this amount of data $\endgroup$ – Madaray Jan 25 '19 at 11:02
0
$\begingroup$

KMeans wasn't the problem, but the silhouette analysis that follows. Python somehow, jump to the silhouette before terminating the fitting of KMeans.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.