2
$\begingroup$

I am using kmeans to cluster some data with 2 features. Not sure I understand why kmeans is producing the clusters I see:

kmeans results

Why would kmeans not cluster these points in a way that matches what we would expect visually looking at the data? Why are seemingly random points in the middle of visual clusters being put into a second cluster like that?

The code I am running:

cols = ['col1', 'col2']
features = map(lambda x: df[x], cols)
input = np.matrix(list(zip(*features)))

scaler = StandardScaler()
scaler.fit(input)
input_scaled = scaler.transform(input)

algo = KMeans(n_clusters=2)
algo.fit(input_scaled)
df['cluster'] = pd.Series(algo.labels_)

sns.lmplot(x=cols[0],y=cols[1],data=df, fit_reg=False, hue='cluster')
$\endgroup$
  • $\begingroup$ For one, your features are on such different scales that the x dimension completely dominates. I suspect it's a problem in how you're determining and plotting the cluster assignments. You should inspect the cluster centers instead $\endgroup$ – Sean Owen Sep 28 '17 at 23:32
  • $\begingroup$ Why would you do kmeans for two features? You can eyeball it with only 2 dimensions $\endgroup$ – Darrin Thomas Sep 29 '17 at 2:53
  • $\begingroup$ It would help if we could reproduce the experiment. Could you provide the list of data-points and link/reference to the KMeans code/library you are utilizing. Besides that, I'd suggest reading the documentation. Typically, kmeans implementations randomly reinitialize a cluster with random assignment(s), when all its points get assigned to another cluster, which might be the problem here. Using Mahalanobis metric instead of Euclidean would help. Initialization of cluster centers? Try multiple runs, if cluster centers disagree a lot then you have a problem. $\endgroup$ – Dynamic Stardust Sep 29 '17 at 5:30
  • $\begingroup$ @SeanOwen I am plotting them with their actual values, but I am using StandardScaler to create the inputs for the algorithm. Does your comment still apply? $\endgroup$ – Ariel Jakobovits Sep 29 '17 at 16:35
  • $\begingroup$ @DarrinThomas Because once my data is labeled by this clustering step, I then plot a bunch of other graphs and display the data points using colors driven by the clustering labels. Helps me determine visually where the clusters fall when breaking the data down differently. Make sense? $\endgroup$ – Ariel Jakobovits Sep 29 '17 at 16:37
0
$\begingroup$

Got it. Thanks to everyone who helped. The issue had nothing to do with kmeans.

I did not realize that when you do this:

dataframe[col] = Series, the series gets merged based on the index, as opposed to a simple appending of the column.

My dataframe had been filtered before I generated the lists of features, so the index was not 0,1,2 but rather 0,2,5, etc. I needed to perform reset_index() on the original dataframe before assigning a new column in the dataframe to the labels from the algorithm.

enter image description here

|improve this answer|||||
$\endgroup$
  • $\begingroup$ There is nothing wrong in accepting your answer as marked and correct. $\endgroup$ – Kiritee Gak Jan 31 '18 at 0:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.