I hope you can help as after several attempts, I'm no longer sure I can get a decent result.
I have a text corpus made of several documents, like the one below (which has been simplified for the sake of the post, bear with me but I didn't test this specifically):
corpus = ["is this good?", "this is bad", "some other text here", "i am hero", "blue jeans", "red carpet", "red dog", "blue sweater", "red hat", "kitty blue"]
I have applied a TD-IDF from sklearn also using the tokenizer parameter to which I have attached my cleaning/parsing text routine. The resulting vector has been ran past to identify the elbow and finally to the KMean model to get the unsupervised clusterization.
vectorizer = TfidfVectorizer(....) X = vectorizer.fit_transform(corpus) model = KMeans(n_clusters=11, random_state=1021) clusters = model.fit_predict(X)
All in all it seems to work, though I may still have doubts on the quality of the output, likely due to the limited numbers of documents (50 in my case).
ISSUE Where I'm struggling is the visualisation part. I'd like to scatterplot both the clusters and the centroids in a unique graph to show somehow where these nodes where and what the center is.
By doing a
cluster_distance = model.transform(X), the resulting 11 dimension array contains a single value that is not of any purpose, because scatterplots require both an X and Y.
If I have a look at X, my 50 X text_features matrix, is just a weight map of terms in the documents.
As far as I can see there is no one method that does what I need.
I therefore moved into considering some sort of data transformation to reduce the dimensionality.
I've tried two different approaches:
- TSNE (from sklearn.manifold import TSNE)
- PCA (from sklearn.decomposition import PCA)
But I have some perplexities.
- With the TSNE I was able only plot the clusters, thought there is not such an evident split (in. my case).
See an extract of the code:
from sklearn.manifold import TSNE tsne_init = 'pca' # could also be 'random' tsne_model = TSNE(n_components=2, random_state=1021, init=tsne_init, perplexity=20, early_exaggeration=4, learning_rate=1000) centroids = model.cluster_centers_ transformed_centroids = tsne_model.fit_transform(centroids) # print(transformed_centroids) # plt.scatter(transformed_centroids[:, 0], transformed_centroids[:, 1], marker='x') Y = tsne_model.fit_transform(X.toarray()) # fitted = tsne_model.fit(Y_sklearn) plt.scatter(Y[:, 0], Y[:, 1], c=predicted_values, s=60, cmap='viridis') plt.scatter(transformed_centroids[:, 0], transformed_centroids[:, 1],c='red', s=150, alpha=0.4)
- With the PCA I was more or less able to get what I want, but to achieve the output, the input has to be fitted and transformed again, hence the cluster assignment changes from the original one.
# FROM https://towardsdatascience.com/k-means-clustering-8e1e64c1561c from sklearn.decomposition import PCA sklearn_pca = PCA(n_components=2) Y_sklearn = sklearn_pca.fit_transform(X.toarray()) fitted = model.fit(Y_sklearn) predicted_values = fitted.predict(Y_sklearn) plt.scatter(Y_sklearn[:, 0], Y_sklearn[:, 1], c=predicted_values, s=60, cmap='viridis') centers = fitted.cluster_centers_ plt.scatter(centers[:, 0], centers[:, 1],c='red', s=150, alpha=0.4)
My questions in using the latter are:
- Would that be ok discharging the results of the original Kmean clusterisation and take the PCA ones (simply because of my visualization needs)?
- Or would that be "safe" plot something but report the clusterization of something different?
Thanks for your help.