4
$\begingroup$

I'm currently learning about clustering. To practice clustering, I am using this dataset.

After running K-means clustering for multiple values of k and plotting the results, I can see that scaling is affecting the results (within-cluster SSE) and I want to use this post to confirm my intuition as to why this is the case.

I don't believe that this is a meaningful reduction in the Within-Cluster SSE because the numerical distances are sensitive to scale, and I don't think that this has any effect on how accurate the model is. Is that intuition correct?

I just wasn't expecting the reduction to be this drastic between standardizing and normalizing.

Code and the results:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('customers.csv')
X = df.iloc[:, [3, 4]].to_numpy()

from sklearn.preprocessing import StandardScaler, MinMaxScaler
ssc, mmsc = StandardScaler(), MinMaxScaler()
X_ssc = ssc.fit_transform(X)
X_mmsc = mmsc.fit_transform(X)

from sklearn.cluster import KMeans
# Unscaled
k_vals = list(range(2, 21))
WCSSE = []
for k in k_vals:
    kmeans = KMeans(n_clusters=k)
    model = kmeans.fit(X)
    WCSSE.append(model.inertia_)
plt.plot(WCSSE, marker='o', markersize=10)

# Standard Scaler
k_vals = list(range(2, 21))
WCSSE = []
for k in k_vals:
    kmeans = KMeans(n_clusters=k)
    model = kmeans.fit(X_ssc)
    WCSSE.append(model.inertia_)
plt.plot(WCSSE, marker='o', markersize=10)

# MinMax scaler
k_vals = list(range(2, 21))
WCSSE = []
for k in k_vals:
    kmeans = KMeans(n_clusters=k)
    model = kmeans.fit(X_mmsc)
    WCSSE.append(model.inertia_)
plt.plot(WCSSE, marker='o', markersize=10)

enter image description here

$\endgroup$
2
  • $\begingroup$ Is there a chance you put axis labels on this and edit your post to express what you mean by 'results' (is it accuracy, for example?)? $\endgroup$ – shepan6 Jul 1 '20 at 8:50
  • $\begingroup$ Sorry about that, updated it now with the labels. By results I was referring to minimization of within cluster SSE, hopefully it's clearer now. $\endgroup$ – fffrost Jul 1 '20 at 9:01
2
$\begingroup$

Is that intuition correct

Yes
There is no improvement in Cluster quality. All the 3 are the same and should be that way.
We can easily observe that all the 3 clusters are forming the elbow at 2.5. Even all other aspects of the 3 plots are exactly the same.

Within Cluster Sum of Squares (WCSS) measures the squared average distance of all the points within a cluster to the center of the cluster.It is the mean distance of each point within the Cluster from the Centroid. No ratio is involved in this metric(i.e. to cancel the impact of scale), hence it will definitely depend on the space size and also on the number of clusters.
Imaging your space is of the size of Earth, then you standardized it to make it as a size of a football, then you make it even smaller i.e. golf ball with Normalization.

I just wasn't expecting the reduction to be this drastic between standardizing and normalizing.

Obviously, the mean distance will reduce in the proportion of Standard deviation in the case of Standardization and "Max value" in the case of Normalization. Having big outliers can have a bigger impact.

Using the same logic, we can see that the metric decreases with the number of clusters too. More cluster means each cluster is closer to its Centrod and hence smaller SSE. That's why it's not a great metric.

You may try to calculate silhouette score, which combines both the Cohesion and Separation for 3 cases

from sklearn.metrics import silhouette_score
silhouette_score(X, kmeans.labels_)

Ref

$\endgroup$
2
  • $\begingroup$ The plots are not exactly the same, nor should they be. The are very very similar, and they should be. If you were scaling the features by equal proportions, the results would be exactly the same, but since StandardScaler and MinMaxScaler will scale the two features by different proportions, each feature's contribution to WCSS will be different depending on the type of scaling. $\endgroup$ – Ben Reiniger Jul 1 '20 at 17:50
  • $\begingroup$ (As an extreme example, suppose one feature is already in [0,1] but the other is uniformly distributed on [0, 10**10]. Without scaling, the first feature is basically ignored by the model optimizing WCSS, but both features are considered important after scaling.) $\endgroup$ – Ben Reiniger Jul 1 '20 at 17:50
2
$\begingroup$

Welcome to the community!

Some points which might help:

  • Clustering, as an unsupervised task, can not be evaluated and usually some external criteria are used to find the best clustering.
  • According to the point above, better to make those assumptions as direct as possible. Starting with EDA (inspecting histograms, plotting boxplots, etc.) gives you a better initial point for analysis. Trying to understand the underlying structure of data directly from decrease of WSCC seems pretty indirect to me thus more difficult. For example a simple EDA tells you if data needs scaling urgently or not. Specially iin your data, ranges are not "dramatically" different, but if you see histogram, you may see, for instance, a exponential feature which needs a log-transform instead of scaling.
  • The last but not the least, the larger the number of clusters get, the smaller the WCSS become. You see it in plots. So the question is if these three scaling show a significant difference in dropping point of WCSS, which they don't and boosts this idea that scaling is not the biggest help here. Otherwise your code is pretty right (and well-written ;) )

I hope it could help to some extent.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you for your comments, it is helpful. You mention the importance of EDA - I am planning to scale up to a much larger dataset and was learning for that purpose. That dataset has high dimensionality, different ranges, and histograms all show quite dramatic positive skew, so I think I'll be transforming and scaling, and exploring alternative evaluations! $\endgroup$ – fffrost Jul 1 '20 at 10:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.