I have more then 100 columns with the values of 1-0. But the two features at the end as seen in the below image, have different values then the rest. Should I rescale the values in the last two columns into a range of [0,1]?

Note : I use K-Means algorithm.

enter image description here


2 Answers 2


Let $x \in \mathbb{R}^{d}$ be a feature vector (one row of your table). Your data has the problem that the distribution per dimension is very different, in particular $x_{1} \in \{0,1\}$ and $x_{d} \in \mathbb{R}$.

Using K-Means means to consider each data point in the euclidean space where the similarity (or distance) is measured. However, the main problem arises when using the euclidean space in your context!

If we measure a distance in the euclidean space for $x,y \in \mathcal{X}$, then $||x-y|| = \sqrt{\sum_{i = 1}^{d}(x_{i}-y_{i})^{2}}$. To simplify the examination, let us now consider the squared euclidean loss $||x-y||^{2} = \sum_{i = 1}^{d}(x_{i}-y_{i})^{2} $ and let $e_{i}:= (x_{i}-y_{i})^2$ be the squared deviation per dimension.

Then $||x-y||^{2} =\sum_{i = 1}^{d}e_{i}$. For a column $i$ with binary values we have $e_{i} \in [0,1]$. However, for the last two columns we have $e_{i} \in \mathbb{R}$.

Therefore, the last two columns have already a higher weight on the distance calculation $||x-y||^{2} =\sum_{i = 1}^{d}e_{i}$, which might not represent the true importance of these dimensions. Even more, by using the euclidean space you implicitly assume that the importance of the features are comparable to each other, which might not be the case.

So what are your options:

1.) Min-Max scaling (so that also the last two dimensions are in $[0,1]$).

You can do that, but you should be aware of the impact: You could say from the perspective of the distance measurement, all dimensions have now equal weight, which might be misleading again.

2.) Standardizing your data

You can do that. It is often better than min-max scaling as you maintain the distribution per dimension. Still you should be aware of the impact: You could say from the perspective of the distance measurement, all dimensions still have kind of an equal weight, which might be misleading again.

3.) You can perform L2 normalization and use the cosine similarity.

It results in the same issues..

Therefore, if the importance of some features is very different, or if the importance of different features cannot be determined, e.g. they are not comparable, using K-Means (or measurements in the euclidean space) is basically a bad idea.

4.) Recall that Random forest is invariant to these issues: If $\mathcal{X}$ is a set of data points that is classified by an inner node of a decision tree that has been trained by the random forest algorithm, then the algorithm provides a threshold $\theta \in \mathbb{R}$ and a dimension $d' \in \{1,\ldots,d\}$. The inner node splits the data into $\{x \in \mathcal{X} \mid x_{d'} \leq \theta \}$ and $\{x \in \mathcal{X} \mid x_{d'} > \theta \}$.

In other words, it learns at each inner node of a decision tree a weak classifier that is applied to a single dimension, thereby avoiding to define a comparable importance weight for each dimension.

You can use a random forest to build a distance measurement (see Wikipedia), and apply K-Means afterwards.

5.) Use K-Means variants.

6.) Use PCA to project your data to a space $\mathbb{R}^{d'}$ with $d' < d$, then apply K-means.

7.) Use an autoencoder to learn a better embedding. Then apply K-means.

  • $\begingroup$ I like this answer. Putting in simple words, if you have some points that look as outliers (even when they aren't) you will have big distortions since k-Means relies on means (averages). $\endgroup$
    – Pithit
    Commented Sep 8, 2020 at 3:09

You should consider standardising all your columns.

K-means creates clusters in a way that minimises the sum of squared distances of points within each of your k (e.g. k = 5) clusters from the mean point in that cluster. It chooses these k mean points (the so-called k means) to minimise the distance. This means that if two of your columns have a much bigger range of values, these are likely to be the main source of error (squared distance from the mean of each cluster) so that your clusters will disproportionately reflect how close your samples were along those axes.

  • $\begingroup$ The thing is some ppl are suggesting that I should normalize last two columns. I am sure I need some change for those columns but not sure if I should do normalization or standardization. And sorry for my lack of knowledge, in either case, do I do for all dataset or only last two columns? $\endgroup$
    – quilliam
    Commented Sep 7, 2020 at 8:35
  • $\begingroup$ Apply your normalisation or standardisation to the whole dataset. You can normalise or standardise. Normalising will only affect the last two columns. Standardising will affect all the columns. I would recommend standardising as it might give better separation between samples where the labels in some columns are imbalanced (e.g. there are more 1s than 0s). $\endgroup$ Commented Sep 7, 2020 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.