When I cluster a lot of data, it is hard to run KMeans and wait it stop until centers has not change, so I have to stop KMeans when it reach maximum number of iterations. Here come problem: how can I evaluate the impact of this KMeans did for my data. I know I can get the inertia_ after KMeans fitting my data to see the sum of distances of samples to their closest cluster center. But How can I get the the inertia_ before KMeans fitting with which I can compare it with the the inertia_ after KMeans fitting, so that I can see the improvement of KMeans did for my data.
How to get the inertia at the begining when using sklearn.cluster.KMeans and MiniBatchKMeans
It sounds like you are grappling with large data set sizes, for which I first suggest switching to
mini-batch k-means. Mini-batch scales better so will be less frustrating.
Regarding apriori estimates of the
inertia_, I suggest using a sample data set to approximate the
inertia_ with appropriate margins of error. But, mini-batch may just preclude your need for apriori
Hope this helps!
$\begingroup$ Thanks! But a little confused about "using a sample data set to approximate the inertia_ with appropriate margins of error". Do you mean that randomly sample from data, and compute inertia_ manually when I set initial centers. Then compute inertia_ when MiniBatchKMeans finished? $\endgroup$– sefira32Oct 18, 2016 at 7:51