Kmeans large dataset

Question

we are currently performing a K-MEANS under scikit-learn on a data set containing 236027 observations with 6 variables in double format (64 bits).

According to our calculations, the complexity of the algorithm is O(n * k * v * i), with n the number of observations, k the number of clusters, v the number of variables and i the number of iterations which is 300 maximum.

Let (236027 * 3 * 6 * 300) * 64 (size of the double in bits): we have 81570931200 bits which is ~10.1 GB of memory.

However, by running the algorithm on a 600 GB VM, it crashed due to lack of memory.

Are we wrong in our calculation of the space required for KMEANS?

Is KMEANS suitable for this volume?

What solutions are available to us?

Thanks you

Answer 1

KMeans wasn't the problem, but the silhouette analysis that follows. Python somehow, jump to the silhouette before terminating the fitting of KMeans.

Answer 2

You should consider hierarchical K-Means Clustering.

Link to show an idea:

https://www.datanovia.com/en/lessons/hierarchical-k-means-clustering-optimize-clusters/

Answer 3

I'm not sure about size of double in python is 64 bits.

You can use mini batch configuration of kmeans: https://scikit-learn.org/stable/modules/clustering.html#mini-batch-kmeans

also maybe DBSCAN will be suitable for you