# Confused by kmeans results

I am using kmeans to cluster some data with 2 features. Not sure I understand why kmeans is producing the clusters I see:

Why would kmeans not cluster these points in a way that matches what we would expect visually looking at the data? Why are seemingly random points in the middle of visual clusters being put into a second cluster like that?

The code I am running:

cols = ['col1', 'col2']
features = map(lambda x: df[x], cols)
input = np.matrix(list(zip(*features)))

scaler = StandardScaler()
scaler.fit(input)
input_scaled = scaler.transform(input)

algo = KMeans(n_clusters=2)
algo.fit(input_scaled)
df['cluster'] = pd.Series(algo.labels_)

sns.lmplot(x=cols[0],y=cols[1],data=df, fit_reg=False, hue='cluster')

• For one, your features are on such different scales that the x dimension completely dominates. I suspect it's a problem in how you're determining and plotting the cluster assignments. You should inspect the cluster centers instead – Sean Owen Sep 28 '17 at 23:32
• Why would you do kmeans for two features? You can eyeball it with only 2 dimensions – Darrin Thomas Sep 29 '17 at 2:53
• It would help if we could reproduce the experiment. Could you provide the list of data-points and link/reference to the KMeans code/library you are utilizing. Besides that, I'd suggest reading the documentation. Typically, kmeans implementations randomly reinitialize a cluster with random assignment(s), when all its points get assigned to another cluster, which might be the problem here. Using Mahalanobis metric instead of Euclidean would help. Initialization of cluster centers? Try multiple runs, if cluster centers disagree a lot then you have a problem. – Dynamic Stardust Sep 29 '17 at 5:30
• @SeanOwen I am plotting them with their actual values, but I am using StandardScaler to create the inputs for the algorithm. Does your comment still apply? – Ariel Jakobovits Sep 29 '17 at 16:35
• @DarrinThomas Because once my data is labeled by this clustering step, I then plot a bunch of other graphs and display the data points using colors driven by the clustering labels. Helps me determine visually where the clusters fall when breaking the data down differently. Make sense? – Ariel Jakobovits Sep 29 '17 at 16:37