What's the difference between finding the average Euclidean distance and using inertia_ in KMeans in sklearn?

Question

I've found two different approaches online when using the Elbow Method to determine the optimal number of clusters for K-Means.

One approach is to use the following code:

distortions_2.append(sum(np.min(cdist(data
                              , kmeanModel.cluster_centers_
                              , 'euclidean')
                              , axis = 1)) / data.shape[0])

Another is to use inertia_ from sklearn.cluster.KMeans:

distortions_3.append(kmeanModel.inertia_)

When I plot the results (using the same random states) both give different results but I'm not sure what the differences are, can anyone help?

Answer 1

K-means does not minimize Euclidean distances, but squared Euclidean distances. This is not the same.

The nearest center is the same for both, but the mean only optimizes the squares. You can find the counterexample on my earlier answers here.

The proper value to use for the basic elbow approach is the "inertia", which is the sum of squares. But some evaluation measures may use other distances. For example the Silhouette index can be used with other distances.

Don't forget that these approaches are just heuristics for trying some k. Always also consider other values.