I have issues finding a way to plot data points colored by cluster with k-means.

I have a very long list of strings.I managed to plot the centroids but not the data points;

import matplotlib.pyplot as plt
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans

vectorizer = TfidfVectorizer(stop_words='english')
X = vectorizer.fit_transform(cleans)

true_k = 5
model = KMeans(n_clusters=true_k, init='k-means++', max_iter=100, n_init=1)
print("Top terms per cluster:")
order_centroids = model.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(true_k):
    print("Cluster %d:" % i),
    for ind in order_centroids[i, :10]:
        print(' %s' % terms[ind]),


plt.scatter(order_centroids[:, 0],order_centroids[:, 1], marker="x", s=150, linewidths=5, zorder=10)

I expect the output to be the data points colored by cluster, and not the centroids

  • $\begingroup$ As in the examples with code in the sklearn documentation? $\endgroup$ Dec 29 '18 at 19:17

Scatter plots work well for 2 dimensional, continuous data. Like the toy examples you see everywhere.

But the data produced by tfidfvectorizer is high-dimensional and sparse.

It is to be expected that almost all points will have a 0 value in any two dimensions you plot. One does not simply plot test as 2d coordinates in a meaningful way. And regular k-means doesn't work that well either. The means just move close together, and all correspond to average stopword vectors - not very useful.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.