Improve results of a clustering

Question

I'm a beginner and I'm trying to do a clustering of a multi-sentence text, but my results are horrible. Any tips for me to improve my result?

import pandas
import pprint
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score
from sklearn.feature_extraction.text import TfidfVectorizer

dataset = pandas.read_csv('text.csv', encoding = 'utf-8')

comments = dataset['comments']

comments_list = comments.values.tolist()

vetorize = TfidfVectorizer()
X = vetorize.fit_transform(comments_list)

clusters_number = 6
model = KMeans(n_clusters = clusters_number, init = 'k-means++', max_iter = 300, n_init = 1)

model.fit(X)

centers = model.cluster_centers_
labels = model.labels_

clusters = {}
for verbatim, label in zip(verbatim_list, labels):
    try:
        clusters[str(label)].append(verbatim)
    except:
        clusters[str(label)] = [verbatim]
pprint.pprint(clusters)

#Top terms for cluster
print("Top termos par cluster:")
ordem_centroides = model.cluster_centers_.argsort()[:, ::-1]
termos = vetorizar.get_feature_names()
for i in range(clusters_number):
    print ("Cluster %d:" % i,)
    for ind in ordem_centroides[i, :10]:
        print (' %s' % termos[ind],)
        print()

I have many different themes present in various clusters. I pre-processed my data (stopwords, lowercase, I removed the punct...). But still I have 'like to cancel order' in one cluster and 'love cancel order' in another. When, in fact, the ideal is to join all 'cancel order' in a single cluster.

Answer 1

The very obvious tip I would say is that K-means is not the algorithm you use for clustering text data in general. The nature of text data is a bit more complicated than structured data on which K-means is one of basic but still working algorithms (of course it also depends on the way you model the text i.e. how you convert a text dataset to a of numbers). Let me propose two things with one hint:

Hint

Document Clustering is also referred to as Topic Modeling. So you really need to have a look at this as I assume you didn't yet (according to using k-means for this problem). Now we see two standard algorithms for topic modeling but I strongly recommend you to see other algorithms as well.

NMF

Non-negative Matrix Factorization is a well-known decomposition method for non-negative matrices like TF-IDF or other variants of Bag-of-Words. You can simply apply it to your problem using SKLearn.

LDA

or Latent Dirichlet Allocation (don't get confused by Linear Discriminant Analysis). This is a pretty standard algorithm for topic modeling and implementations can be found in SKLearn, Gensim, NLTK, Spacy and other NLP/ML libraries.

Hope it helped! Good Luck :)