5
$\begingroup$

8 clusters from k-means I am working on a clustering problem. I have 11 features. My complete data frame has 70-80% zeros. The data had outliers that I capped at 0.5 and 0.95 percentile. However, I tried k-means (python) on data and received a very unusual cluster that looks like a cuboid. I am not sure if this result is really a cluster or has something gone wrong?

The main reason for my worry, why is it looking like a cuboid and why are the axes orthogonal?

one thing to notice is that: I first reduced the dimensionality using PCA to two dimensions and performed clustering on the same and the plot here is on the 2-dim PCA data

Edit : I chose k using silhouette index in python.

$\endgroup$
  • $\begingroup$ What langage / code did you use ? How did you select k ? Where do your data come from ? $\endgroup$ – lcrmorin Jan 24 at 20:25
  • $\begingroup$ @Icrmorin Edited the question. Please have a look. Data is private, so won't be able to share much details about the data. $\endgroup$ – Akash Dubey Jan 24 at 20:28
  • $\begingroup$ Before doing k-means did you do a basic EDA ? (like mentionned here : datascience.stackexchange.com/questions/66905/…) Is it possible that your data are on a cuboid ? (K means won't change the structure of your data). $\endgroup$ – lcrmorin Jan 24 at 20:32
  • $\begingroup$ @Icrmorin No, It's not possible that my data is on cuboid. The shape of the cuboid changes a bit (but remains orthogonal) when I increase or decrease the capping. What kind of EDA are you suggesting ? $\endgroup$ – Akash Dubey Jan 24 at 20:44
  • $\begingroup$ As per my answer, the cuboid pattern appears to be linked to your capping. That's not bad per se. However that does not begin to answer the question about what you are trying to do with your K-means. $\endgroup$ – lcrmorin Jan 24 at 21:11
2
$\begingroup$

K-means don't modify the underlying structure of your data. K-means will just provide the 'color' part of your graph.

To answer the question about why do you get a cuboid, it's because your underlying data are a cuboid. Not necessarily by construction, but that's what happen when you cap your data. As an exemple, look at the following code :

X1 = c(rnorm(1000))
X2 = c(rnorm(1000))
q95_1 = quantile(X1,0.95)
q95_2 = quantile(X2,0.95)
q5_1 = quantile(X1,0.05)
q5_2 = quantile(X2,0.05)
X1[X1>q95_1]=q95_1
X2[X2>q95_2]=q95_2
X1[X1<q5_1]=q5_1
X2[X2<q5_2]=q5_2
plot(X1,X2)

The code simulate two random gaussian and cap them at 5% and 95%.

this is what you get :

enter image description here

Notice the squaroid pattern ? This is why you get a cuboid in 3D.

Ps: I can't help but say that's what you get when you do k-means without properly looking at your variables (see: What value can I gain by doing exploratory data analysis on features (and thus data) before doing clustering? for an infinite loop).

$\endgroup$
  • $\begingroup$ @ What kind of EDA and what to look for in the variables ? Else it is like searching a pearl in a sea. $\endgroup$ – Akash Dubey Jan 24 at 21:28
  • $\begingroup$ Well just plotting your data before k-means would have shown you the same cuboid data structure and answered your question : the cuboid data structure does not come from the k-means, but the data. I upvoted your question because that cuboid was rather intringuing. If the answer suits you and show some effort (a bit of code for exemple), may I ask you to accept / upvote it ? $\endgroup$ – lcrmorin Jan 24 at 21:33
  • $\begingroup$ I did plot the data and by plotting data what I mean is I plotted for every feature for its distribution and histogram individually. I did not find a cuboid though and How would I ? I don't have any way to plot the whole data at once but look for every feature. $\endgroup$ – Akash Dubey Jan 24 at 21:37
  • $\begingroup$ Well EDA should be done often, if not at every step, that means after capping to see what capping achieved, in 2D/3D before clustering to see if it will work or not. Hope that answer your other question. $\endgroup$ – lcrmorin Jan 24 at 21:40
  • $\begingroup$ one thing to notice is that: I first reduced the dimensionality using PCA to two dimensions and performed clustering on the same and the plot here is on the 2-dim PCA data. $\endgroup$ – Akash Dubey Jan 25 at 4:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.