Can distortion be derived from inertia rather than recalculating it from scratch in case of kmeans?

Question

I got this definitional difference between distortion and inertia from here:

Two values are of importance here — distortion and inertia. Distortion is the average of the euclidean squared distance from the centroid of the respective clusters. Inertia is the sum of squared distances of samples to their closest cluster centre

However, when I searched for an example from here:

distortions.append(sum(np.min(cdist(X, kmeanModel.cluster_centers_,
                                        'euclidean'), axis=1)) / X.shape[0])

I took one inertia value from:

and 1 distortion value from :

however the values 217.64 and 3.45 aren't simply a division of 217.64 by count of variables in x (17).

Can anyone detail on what exactly is the formula for distortion?

Answer 1

The reason is that

distortions.append(sum(np.min(cdist(X, kmeanModel.cluster_centers_,
                                        'euclidean'), axis=1)) / X.shape[0])

does not square the distance as stated in the definition

Distortion is the average of the euclidean squared distance from the centroid of the respective clusters

If you replace

    mapping1[k] = sum(np.min(cdist(X, kmeanModel.cluster_centers_,
                                   'euclidean'), axis=1)) / X.shape[0]

with

    mapping_squared[k] = sum(np.square(np.min(cdist(X, kmeanModel.cluster_centers_,
                                   'euclidean'), axis=1))) / X.shape[0]

you will get the desired result:

IN: for key, val in mapping_squared.items():
    print(f'{key} : {val}')

OUT:  
1 : 12.802768166089965
2 : 4.025210084033612
3 : 0.954621848739496
4 : 0.7467787114845938
5 : 0.5647058823529413
6 : 0.42156862745098034
7 : 0.29901960784313725
8 : 0.24019607843137258
9 : 0.16666666666666669

That is, the values in mapping_squared can be multiplied by the number of samples to match inertia:

IN: np.array(distortions_square) * X.shape[0]

OUT:
array([217.64705882,  68.42857143,  16.22857143,  12.6952381 ,
         9.66666667,   7.51666667,   5.66666667,   4.08333333,
         3.        ])

On a side note: Distortion and SSE are usually used interchangeably. See, for example, the paper Scaling Clustering Algorithms to Large Databases:

Distortion is the sum of the L2 distances squared between the data items and the mean of their assigned cluster

This question touches the subject too.