When I cluster a lot of data, it is hard to run KMeans and wait it stop until centers has not change, so I have to stop KMeans when it reach maximum number of iterations. Here come problem: how can I evaluate the impact of this KMeans did for my data. I know I can get the inertia_ after KMeans fitting my data to see the sum of distances of samples to their closest cluster center. But How can I get the the inertia_ before KMeans fitting with which I can compare it with the the inertia_ after KMeans fitting, so that I can see the improvement of KMeans did for my data.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
It sounds like you are grappling with large data set sizes, for which I first suggest switching to mini-batch k-means. Mini-batch scales better so will be less frustrating.
Regarding apriori estimates of the inertia_, I suggest using a sample data set to approximate the inertia_ with appropriate margins of error. But, mini-batch may just preclude your need for apriori inertia_.
Hope this helps!
-
$\begingroup$ Thanks! But a little confused about "using a sample data set to approximate the inertia_ with appropriate margins of error". Do you mean that randomly sample from data, and compute inertia_ manually when I set initial centers. Then compute inertia_ when MiniBatchKMeans finished? $\endgroup$– sefira32Oct 18, 2016 at 7:51