When I cluster a lot of data, it is hard to run KMeans and wait it stop until centers has not change, so I have to stop KMeans when it reach maximum number of iterations. Here come problem: how can I evaluate the impact of this KMeans did for my data. I know I can get the inertia_ after KMeans fitting my data to see the sum of distances of samples to their closest cluster center. But How can I get the the inertia_ before KMeans fitting with which I can compare it with the the inertia_ after KMeans fitting, so that I can see the improvement of KMeans did for my data.
It sounds like you are grappling with large data set sizes, for which I first suggest switching to
mini-batch k-means. Mini-batch scales better so will be less frustrating.
Regarding apriori estimates of the
inertia_, I suggest using a sample data set to approximate the
inertia_ with appropriate margins of error. But, mini-batch may just preclude your need for apriori
Hope this helps!