1
$\begingroup$

When I cluster a lot of data, it is hard to run KMeans and wait it stop until centers has not change, so I have to stop KMeans when it reach maximum number of iterations. Here come problem: how can I evaluate the impact of this KMeans did for my data. I know I can get the inertia_ after KMeans fitting my data to see the sum of distances of samples to their closest cluster center. But How can I get the the inertia_ before KMeans fitting with which I can compare it with the the inertia_ after KMeans fitting, so that I can see the improvement of KMeans did for my data.

$\endgroup$
1
$\begingroup$

It sounds like you are grappling with large data set sizes, for which I first suggest switching to mini-batch k-means. Mini-batch scales better so will be less frustrating.

Regarding apriori estimates of the inertia_, I suggest using a sample data set to approximate the inertia_ with appropriate margins of error. But, mini-batch may just preclude your need for apriori inertia_.

Hope this helps!

$\endgroup$
1
  • $\begingroup$ Thanks! But a little confused about "using a sample data set to approximate the inertia_ with appropriate margins of error". Do you mean that randomly sample from data, and compute inertia_ manually when I set initial centers. Then compute inertia_ when MiniBatchKMeans finished? $\endgroup$
    – sefira32
    Oct 18 '16 at 7:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.