2
$\begingroup$

I'm a beginner and I'm trying to do a clustering of a multi-sentence text, but my results are horrible. Any tips for me to improve my result?

import pandas
import pprint
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score
from sklearn.feature_extraction.text import TfidfVectorizer

dataset = pandas.read_csv('text.csv', encoding = 'utf-8')

comments = dataset['comments']

comments_list = comments.values.tolist()

vetorize = TfidfVectorizer()
X = vetorize.fit_transform(comments_list)

clusters_number = 6
model = KMeans(n_clusters = clusters_number, init = 'k-means++', max_iter = 300, n_init = 1)

model.fit(X)

centers = model.cluster_centers_
labels = model.labels_

clusters = {}
for verbatim, label in zip(verbatim_list, labels):
    try:
        clusters[str(label)].append(verbatim)
    except:
        clusters[str(label)] = [verbatim]
pprint.pprint(clusters)

#Top terms for cluster
print("Top termos par cluster:")
ordem_centroides = model.cluster_centers_.argsort()[:, ::-1]
termos = vetorizar.get_feature_names()
for i in range(clusters_number):
    print ("Cluster %d:" % i,)
    for ind in ordem_centroides[i, :10]:
        print (' %s' % termos[ind],)
        print()

I have many different themes present in various clusters. I pre-processed my data (stopwords, lowercase, I removed the punct...). But still I have 'like to cancel order' in one cluster and 'love cancel order' in another. When, in fact, the ideal is to join all 'cancel order' in a single cluster.

$\endgroup$
  • 1
    $\begingroup$ why do you say your results are terrible? What is your goal? Can you show some examples of what your text.csv looks like and what kind of information you expect to get from clustering? $\endgroup$ – Bruno Lubascher Jul 27 '18 at 10:36
  • $\begingroup$ I have many different themes present in various clusters. I pre-processed my data (stopwords, lowercase, I removed the punct...). But still I have 'like to cancel order' in one cluster and 'love cancel order' in another. When, in fact, the ideal is to join all 'cancel order' in a single cluster. $\endgroup$ – marin Jul 27 '18 at 10:49
  • 1
    $\begingroup$ Ok, why did you choose 6 clusters? if you choose 6 but there in truth are 2, you will see your results split. Do you have a predefined number of clusters that you need to find the items that belong to it? $\endgroup$ – Bruno Lubascher Jul 27 '18 at 11:35
  • $\begingroup$ I would like to be at least 30. But I do not know if it is possible. I think my program is really bad because even with 60 (yes, 60!) clusters, I have errors. $\endgroup$ – marin Jul 27 '18 at 12:04
3
$\begingroup$

The very obvious tip I would say is that K-means is not the algorithm you use for clustering text data in general. The nature of text data is a bit more complicated than structured data on which K-means is one of basic but still working algorithms (of course it also depends on the way you model the text i.e. how you convert a text dataset to a of numbers). Let me propose two things with one hint:

Hint

Document Clustering is also referred to as Topic Modeling. So you really need to have a look at this as I assume you didn't yet (according to using k-means for this problem). Now we see two standard algorithms for topic modeling but I strongly recommend you to see other algorithms as well.

NMF

Non-negative Matrix Factorization is a well-known decomposition method for non-negative matrices like TF-IDF or other variants of Bag-of-Words. You can simply apply it to your problem using SKLearn.

LDA

or Latent Dirichlet Allocation (don't get confused by Linear Discriminant Analysis). This is a pretty standard algorithm for topic modeling and implementations can be found in SKLearn, Gensim, NLTK, Spacy and other NLP/ML libraries.

Hope it helped! Good Luck :)

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Something I forgot: 1) A bit of cleaning works. Use NLTK.word_tokenize for tokenization. It cleans punctuations and stuff. 2) Include n-grams in your analysis $\endgroup$ – Kasra Manshaei Jul 27 '18 at 12:59
  • 1
    $\begingroup$ You should edit your answer to add this information. $\endgroup$ – Mephy Jul 27 '18 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.