Why the Silhouette Score and optimal number of Cluster changes when using 2D and 3D data?

Question

I am experimenting with Kmeans clustering. My data (vectors) was in 300 dimensions which I am converting into 2D and 3D using PCA. Now, to find the optimal number of clusters, I used the Silhouette score. However, for 2D the best Silhouette score is showing for 3 clusters (silhouette score = 0.45), and for 3D it is showing 9 clusters (silhouette score = 0.3861).

I want to know whether it is normal? If yes, what is the reason for this? What should I choose 2D or 3D?

Also, the reason for experimenting with 2D and 3D is because I wanted to plot the 3D graph using seaborn.

Answer 1

Yes it can happen. In fact it is quite normal since there are different clusters in 2D and different in 3D, since more or less information is added to data (by having more dimensions). This is a by-product of the curse of dimensionality.

Adding as more relevant information as possible would make clusters more close to underlying groups. So 3D would be better than 2D. This is a general observation. There are of course cases where projecting data in a low-dimensionalal manifold is indeed better since it can eliminate noise and/or capture specific attributes better than clustering on all (possibly irrelevant) dimensions (another by-product of the curse of dimensionality).

If the relevant information in your data has low dimensionality but this information is correlated along many dimensions in the original data then a feature extraction method is needed in order to capture the low-dimensional relevant information from original data (eg PCA, ICA ,..).

For some references along this direction see for example:

1. How to cluster in High Dimensions
2. An investigation of K-means clustering to high and multi-dimensional biological data
3. How do I know my k-means clustering algorithm is suffering from the curse of dimensionality?