k-means: Only one-dimensional cluster predictions in two-dimensional space

Question

For this dataset, it seems that the predictions of my k-means model only consider the horizontal axis, although the cluster centers seem reasonable.

Is something wrong with this classification? Please note the color of the grid in the background.

I use scikit-learn, here is the code fragment of classification and visualization.

model = KMeans(n_clusters = 5)    
model.fit(df_stuff[['Stuff','Other Stuff']])

fig = plt.figure()
ax = fig.add_axes([0,0,1,1])

ax.scatter(df_stuff['Stuff'], df_stuff['Other Stuff'],c=model.labels_,s=80,cmap='rainbow')
ax.set_xlabel('Stuff')
ax.set_ylabel('Other Stuff')
ax.set_title('Strange Clusters')

# Draw Cluster Centers
for center in model.cluster_centers_:
    ax.scatter(center[0],center[1],c='black',s=5120,alpha=0.2)

# Draw Cluster Grid
cluster_grid = {'x': [], 'y': [], 'cluster': []}
for x in np.linspace(df_stuff['Stuff'].min(),df_stuff['Stuff'].max(),25):
    for y in np.linspace(0.35,0.6,25):
        cluster_grid['x'].append(x)
        cluster_grid['y'].append(y)
        cluster_grid['cluster'].append(model.predict([[x,y]])[0])

ax.scatter(cluster_grid['x'],cluster_grid['y'],c=cluster_grid['cluster'],cmap='rainbow',alpha=0.4,s=10)

Answer 1

You are overfitting your data. You are fitting 5 clusters for ~20 data points. The red and blue clusters only have a single data point. Either get more data or fit fewer clusters.

The Elbow method will help decide how many clusters are appropriate.

Answer 2

KMeans does correctly do what it is supposed to do.

Just plot your data correctly, with the same scale on both axes...

Y deviations do not matter, they are tiny compared to the X axis. Deviations there are 100x larger, so squared deviations even 10000x. Since KMeans minimized squared errors, only x matters

When plotted correctly, your data more looks like this:

Answer 3

It might not be your clustering that is the problem. But the visual representation of your clustering. It is unclear why you are plotting elements on a hardcoded grid. You should just plot the actual values of the raw data. Something like:

import matplotlib.pyplot as plt

# Plot the data
X = df_stuff[['Stuff','Other Stuff']]
plt.scatter(X[:, 0], X[:, 1], c=model.predict(X))

# Plot k-means clusters centers
centers = kmeans.cluster_centers_
plt.scatter(centers[:, 0], centers[:, 1], c='black')