Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have more then 100 columns with the values of 1-0. But the two features at the end as seen in the below image, have different values then the rest. Should I rescale the values in the last two columns into a range of [0,1]?
Note : I use K-Means algorithm.
Let $x \in \mathbb{R}^{d}$ be a feature vector (one row of your table). Your data has the problem that the distribution per dimension is very different, in particular $x_{1} \in \{0,1\}$ and $x_{d} \in \mathbb{R}$.
Using K-Means means to consider each data point in the euclidean space where the similarity (or distance) is measured. However, the main problem arises when using the euclidean space in your context!
If we measure a distance in the euclidean space for $x,y \in \mathcal{X}$, then $||x-y|| = \sqrt{\sum_{i = 1}^{d}(x_{i}-y_{i})^{2}}$. To simplify the examination, let us now consider the squared euclidean loss $||x-y||^{2} = \sum_{i = 1}^{d}(x_{i}-y_{i})^{2} $ and let $e_{i}:= (x_{i}-y_{i})^2$ be the squared deviation per dimension.
Then $||x-y||^{2} =\sum_{i = 1}^{d}e_{i}$. For a column $i$ with binary values we have $e_{i} \in [0,1]$. However, for the last two columns we have $e_{i} \in \mathbb{R}$.
Therefore, the last two columns have already a higher weight on the distance calculation $||x-y||^{2} =\sum_{i = 1}^{d}e_{i}$, which might not represent the true importance of these dimensions. Even more, by using the euclidean space you implicitly assume that the importance of the features are comparable to each other, which might not be the case.
So what are your options:
1.) Min-Max scaling (so that also the last two dimensions are in $[0,1]$).
You can do that, but you should be aware of the impact: You could say from the perspective of the distance measurement, all dimensions have now equal weight, which might be misleading again.
2.) Standardizing your data
You can do that. It is often better than min-max scaling as you maintain the distribution per dimension. Still you should be aware of the impact: You could say from the perspective of the distance measurement, all dimensions still have kind of an equal weight, which might be misleading again.
3.) You can perform L2 normalization and use the cosine similarity.
It results in the same issues..
Therefore, if the importance of some features is very different, or if the importance of different features cannot be determined, e.g. they are not comparable, using K-Means (or measurements in the euclidean space) is basically a bad idea.
4.) Recall that Random forest is invariant to these issues: If $\mathcal{X}$ is a set of data points that is classified by an inner node of a decision tree that has been trained by the random forest algorithm, then the algorithm provides a threshold $\theta \in \mathbb{R}$ and a dimension $d' \in \{1,\ldots,d\}$. The inner node splits the data into $\{x \in \mathcal{X} \mid x_{d'} \leq \theta \}$ and $\{x \in \mathcal{X} \mid x_{d'} > \theta \}$.
In other words, it learns at each inner node of a decision tree a weak classifier that is applied to a single dimension, thereby avoiding to define a comparable importance weight for each dimension.
You can use a random forest to build a distance measurement (see Wikipedia), and apply K-Means afterwards.
5.) Use K-Means variants.
6.) Use PCA to project your data to a space $\mathbb{R}^{d'}$ with $d' < d$, then apply K-means.
7.) Use an autoencoder to learn a better embedding. Then apply K-means.
You should consider standardising all your columns.
K-means creates clusters in a way that minimises the sum of squared distances of points within each of your k (e.g. k = 5) clusters from the mean point in that cluster. It chooses these k mean points (the so-called k means) to minimise the distance. This means that if two of your columns have a much bigger range of values, these are likely to be the main source of error (squared distance from the mean of each cluster) so that your clusters will disproportionately reflect how close your samples were along those axes.
