20
$\begingroup$

Is it necessary to standardize your data before cluster? In the example from scikit learn about DBSCAN, here they do this in the line:

X = StandardScaler().fit_transform(X)

But I do not understand why it is necessary. After all, clustering does not assume any particular distribution of data - it is an unsupervised learning method so its objective is to explore the data.

Why would it be necessary to transform the data?

$\endgroup$
  • $\begingroup$ Thank you for the detailed responses! As a follow-up to all of your answers: I understand that if the features are on different scales, this could present a problem. However, if the distance metric is normalized to the variance, does this achieve the same result as standard scaling before clustering? i.e. I usually use a normalized euclidean distance related - does this also mitigate scaling effects? $\endgroup$ – Candic3 Aug 7 '15 at 16:38
18
$\begingroup$

Normalization is not always required, but it rarely hurts.

Some examples:

K-means:

K-means clustering is "isotropic" in all directions of space and therefore tends to produce more or less round (rather than elongated) clusters. In this situation leaving variances unequal is equivalent to putting more weight on variables with smaller variance.

Example in Matlab:

X = [randn(100,2)+ones(100,2);...
     randn(100,2)-ones(100,2)];

% Introduce denormalization
% X(:, 2) = X(:, 2) * 1000 + 500;

opts = statset('Display','final');

[idx,ctrs] = kmeans(X,2,...
                    'Distance','city',...
                    'Replicates',5,...
                    'Options',opts);

plot(X(idx==1,1),X(idx==1,2),'r.','MarkerSize',12)
hold on
plot(X(idx==2,1),X(idx==2,2),'b.','MarkerSize',12)
plot(ctrs(:,1),ctrs(:,2),'kx',...
     'MarkerSize',12,'LineWidth',2)
plot(ctrs(:,1),ctrs(:,2),'ko',...
     'MarkerSize',12,'LineWidth',2)
legend('Cluster 1','Cluster 2','Centroids',...
       'Location','NW')
title('K-means with normalization')

enter image description here

enter image description here

(FYI: How can I detect if my dataset is clustered or unclustered (i.e. forming one single cluster)

Distributed clustering:

The comparative analysis shows that the distributed clustering results depend on the type of normalization procedure.

Artificial neural network (inputs):

If the input variables are combined linearly, as in an MLP, then it is rarely strictly necessary to standardize the inputs, at least in theory. The reason is that any rescaling of an input vector can be effectively undone by changing the corresponding weights and biases, leaving you with the exact same outputs as you had before. However, there are a variety of practical reasons why standardizing the inputs can make training faster and reduce the chances of getting stuck in local optima. Also, weight decay and Bayesian estimation can be done more conveniently with standardized inputs.

Artificial neural network (inputs/outputs)

Should you do any of these things to your data? The answer is, it depends.

Standardizing either input or target variables tends to make the training process better behaved by improving the numerical condition (see ftp://ftp.sas.com/pub/neural/illcond/illcond.html) of the optimization problem and ensuring that various default values involved in initialization and termination are appropriate. Standardizing targets can also affect the objective function.

Standardization of cases should be approached with caution because it discards information. If that information is irrelevant, then standardizing cases can be quite helpful. If that information is important, then standardizing cases can be disastrous.


Interestingly, changing the measurement units may even lead one to see a very different clustering structure: Kaufman, Leonard, and Peter J. Rousseeuw.. "Finding groups in data: An introduction to cluster analysis." (2005).

In some applications, changing the measurement units may even lead one to see a very different clustering structure. For example, the age (in years) and height (in centimeters) of four imaginary people are given in Table 3 and plotted in Figure 3. It appears that {A, B ) and { C, 0) are two well-separated clusters. On the other hand, when height is expressed in feet one obtains Table 4 and Figure 4, where the obvious clusters are now {A, C} and { B, D}. This partition is completely different from the first because each subject has received another companion. (Figure 4 would have been flattened even more if age had been measured in days.)

To avoid this dependence on the choice of measurement units, one has the option of standardizing the data. This converts the original measurements to unitless variables.

enter image description here enter image description here

Kaufman et al. continues with some interesting considerations (page 11):

From a philosophical point of view, standardization does not really solve the problem. Indeed, the choice of measurement units gives rise to relative weights of the variables. Expressing a variable in smaller units will lead to a larger range for that variable, which will then have a large effect on the resulting structure. On the other hand, by standardizing one attempts to give all variables an equal weight, in the hope of achieving objectivity. As such, it may be used by a practitioner who possesses no prior knowledge. However, it may well be that some variables are intrinsically more important than others in a particular application, and then the assignment of weights should be based on subject-matter knowledge (see, e.g., Abrahamowicz, 1985). On the other hand, there have been attempts to devise clustering techniques that are independent of the scale of the variables (Friedman and Rubin, 1967). The proposal of Hardy and Rasson (1982) is to search for a partition that minimizes the total volume of the convex hulls of the clusters. In principle such a method is invariant with respect to linear transformations of the data, but unfortunately no algorithm exists for its implementation (except for an approximation that is restricted to two dimensions). Therefore, the dilemma of standardization appears unavoidable at present and the programs described in this book leave the choice up to the user.

$\endgroup$
  • 1
    $\begingroup$ This answer is phenomenal $\endgroup$ – Candic3 Jan 21 '17 at 0:06
5
$\begingroup$

Standardizing data is recommended because otherwise the range of values in each feature will act as a weight when determining how to cluster data, which is typically undesired.

For example consider the standard metric for most clustering algorithms (including DBSCAN in sci-kit learn) -- euclidean, otherwise known as the L2 norm. If one of your features has a range of values much larger than the others, clustering will be completely dominated by that one feature. To illustrate this look at the simple example below:

>>> import numpy as np
>>> from sklearn.preprocessing import StandardScaler
>>> from sklearn.metrics.pairwise import euclidean_distances
>>> 
>>> X = np.array([[1,2,100],[4,3,50],[1,1,75]])
>>> 
>>> print X
[[  1   2 100]
 [  4   3  50]
 [  1   1  75]]
>>> 
>>> print euclidean_distances(X)
[[  0.          50.0999002   25.01999201]
 [ 50.0999002    0.          25.25866188]
 [ 25.01999201  25.25866188   0.        ]]
>>> print euclidean_distances(StandardScaler().fit_transform(X))
[[ 0.          3.46410162  1.73205081]
 [ 3.46410162  0.          3.46410162]
 [ 1.73205081  3.46410162  0.        ]]

From this you should see that the euclidean distances between the non-standardized versions are dominated by the third column because its range of values is much larger than the other two. However when the data is standardized this no longer becomes an issue and weights each feature as being equal when calculating the distance between each data point.

$\endgroup$
1
$\begingroup$

It isn't strictly necessary to standardise, whether it is required or not may depend on the distance metric you choose.

For instance if you opt for the Mahalanobis distance then separation will be based upon the number of standard deviations separating points and not the absolute distance between them, as such it is a scale invariant metric.

As with many things in machine learning there is no hard and fast answer and the only way to know for sure is to apply a variety of techniques see which gives the most appropriate results for your data.

$\endgroup$
0
$\begingroup$

I found in some cases useful to define a "business evaluation" function, defining the "importance" of the dimensions used for clustering. E.g. for greengrocer clustering the customers, if apples are twice as expensive as oranges, the number of apples will be doubled.

$\endgroup$
  • $\begingroup$ But, then couldn't you just include that as an additional feature that you cluster on? i.e. the price becomes a feature to cluster on? $\endgroup$ – Candic3 Aug 7 '15 at 16:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.