1
$\begingroup$

I am running a K-means algorithm (using the sklearn implementation) on an aggregated dataset of ~350k datapoints on a 6 dimension hyper-plane (using 6 features).

I would like to do the same but in the "non-aggregated" version of my dataset, which is ~1b datapoints using the same 6 features

I know this is a very heavy task for K-means, the number of datapoints is just too big, even though the dimensions' size is pretty small.

Are there any suggestions of other algorithms that would help me on this task, apart from mini batch K-means ?

$\endgroup$
1
$\begingroup$

I don't know if there are other clustering methods which would work with this amount of data, but with K-means I would suggest this:

  1. Run K-means with a varying number of instances picked randomly and study how much variation there is between the centroids depending on the data size (you can also study the variation across different random samples). I would expect the centroids to stabilize quite quickly with respect to data size: assuming the centroids become stable with data size N, there's no point running the full K-means process with more data.
  2. Having obtained a model with N data points (the centroids), the model can be applied to all the remaining data points in order to find which cluster they belong to.

This is much more efficient than running the K-means process over the whole data since in the second stage the centroids are fixed, the algorithm doesn't have to iteratively update them.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.