0
$\begingroup$

I am trying to classify texts using kmeans, TfidfVectorizer, PCA. However, it seems that many texts are not correctly classified as you can see: I have texts in cluster2 that should be in Cluster 0 or 1. My question is on how to improve the results, if increasing the number of clusters or adding more constraints (like specific words to look at for clustering texts).

Help and suggestions will be greatly appreciated.

1

Code: 

def preprocessing(line):
    line = re.sub(r"[^a-zA-Z]", " ", line.lower())
    words = word_tokenize(line)
    words_lemmed = [WordNetLemmatizer().lemmatize(w) for w in words if w not in stop_words]
    return words_lemmed


vect =TfidfVectorizer(tokenizer=preprocessing)
vectorized_text=vect.fit_transform(df['Text'])
kmeans =KMeans(n_clusters=n).fit(vectorized_text)


cl=kmeans.predict(vectorized_text)
df['Predicted']=pd.Series(cl, index=df.index)

df.groupby("Predicted").count()

kmeans_labels =KMeans(n_clusters=n).fit(vectorized_text).labels_

pipeline = Pipeline([('tfidf', TfidfVectorizer())])
X = pipeline.fit_transform(df['Text']).todense()
pca = PCA(n_components=n).fit(X)
data2D = pca.transform(X)


kmeans.fit(X)
centers2D = pca.transform(kmeans.cluster_centers_)
labels=kmeans.labels_
cluster_name = ["Cluster"+str(i) for i in set(labels)]
Improve this question
$\endgroup$
1
$\begingroup$

You have tagged this as text-classification, but describe your efforts to cluster the documents. My answer is based on the assumption that you don't know how the documents are grouped for you to classify new documents. The first step would be to determine the clusters/classes/groups and then classify new documents.

There are at least 2 things one can think of for this problem.

  1. Clustering Algorithm
  2. Data for the clustering algorithm

1. Clustering Algorithm

K-Means clustering is a simple and fast algorithm that produces adequate results. It is almost always the first tool that people reach for, but it has some limitations a) User must choose the number of clusters ahead of running it b) Data must not be very sparse c) Data must be reasonably distributed around the centroid - spherical in nature

See also, k-Means Advantages and Disadvantages.

Do you know apriori, how many clusters you have? You could experiment by running the algorithm multiple times with different cluster sizes and observing the silhouette score. Then choose the one with the best score or one that fits your expectations the best.

There are other algorithms that might suit your needs better, like DBSCAN or Agglomerative clustering. They don't require the user to choose number of parameters, but do require you to pass in metrics to determine cluster separation. I've used DBSCAN effectively, but don't have as much experience with Agglomerative clustering. Once again, you may have to experiment with the _min_samples_ and eps parameters for DBSCAN. SKlearn has good description in the Clustering User Guide.

2. Data for clustering algorithm

TF-IDF is a good metric for the text and it is a simple and quick way to produce. It is a representation of the different terms that occur with the documents and the weight associated with it. You have a few different options to try out

  1. sklearn.TfidfVectorizer can take a custom vocabulary. This will help by reducing the vocabulary to important terms. Also, as suggested by @Graph4Me, you should eliminate stopwords. Both of these techniques will reduce the sparsity of the matrix, in turn making your clustering algorithms more efficient.

  2. Change from a TF-IDF model to use an word embedding. This is much more complex, but you could use GloVe or BERT to produce them. Spacy can also produce an embedding vector.This will give you an contextual representation rather than just a count/density based on presence of words.

Improve this answer
$\endgroup$
1
$\begingroup$

It can be improved in many ways. However, the best start would be if you elaborate on your pipeline together with choosen parameters, as well as your data.

Improve this answer
$\endgroup$
10
  • $\begingroup$ Thank you Graph4Me. Could you please give me some tips (or just an example) on that (on parameters to choose) and what you mean with elaborate my data? The only thing that I have done was to remove stopwords from texts. Many thanks $\endgroup$ – LdM Jun 6 '20 at 15:01
  • $\begingroup$ Among other things, the following details are missing: Do you apply Kmeans on TfidfVectorizer or do you apply TfidfVectorizer, then PCA, then K-means ? What is your selected K. How exactly do you use TfidfVectorizer? What are you vectors? In which way should the text in the clusters be similar? $\endgroup$ – Graph4Me Consultant Jun 6 '20 at 16:23
  • $\begingroup$ Also note, of course you can apply word2vec other more modern techniques, but if you really want to learn something, I would not recommend just plug-in some tools at the beginning but to understand why the current method fails.. $\endgroup$ – Graph4Me Consultant Jun 6 '20 at 16:27
  • $\begingroup$ Do you have labels? A training set of documents where you know how they should be clustered? $\endgroup$ – Graph4Me Consultant Jun 6 '20 at 16:32
  • $\begingroup$ I updated the question including code. @Graph4Me $\endgroup$ – LdM Jun 6 '20 at 17:48
1
$\begingroup$

Try to use the concatenated vector of average word2vec and max pool word2vec. You may get better results than tfidf and PCA. Remove the words like stop words. Also, try with lemmatized words.
If you okay with computational time, go for transformer-based transfer learning embedding like BERT, ALBERT etc...

Improve this answer
$\endgroup$
0
$\begingroup$

Different approaches can be applied. In addition to the other answers, you can check: http://brandonrose.org/clustering#Tf-idf-and-document-similarity

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.