we are currently performing a K-MEANS under scikit-learn on a data set containing 236027 observations with 6 variables in double format (64 bits).
According to our calculations, the complexity of the algorithm is O(n * k * v * i), with n the number of observations, k the number of clusters, v the number of variables and i the number of iterations which is 300 maximum.
Let (236027 * 3 * 6 * 300) * 64 (size of the double in bits): we have 81570931200 bits which is ~10.1 GB of memory.
However, by running the algorithm on a 600 GB VM, it crashed due to lack of memory.
Are we wrong in our calculation of the space required for KMEANS?
Is KMEANS suitable for this volume?
What solutions are available to us?