I have a column that contains all texts that I would like to cluster in order to find some patterns/similarity among each other.

Word2vec is a two-layer neural net that processes text by “vectorizing” words. 

Its input is a text corpus and its output is a set of vectors: feature vectors that represent words in that corpus. 

While Word2vec is not a deep neural network, it turns text into a numerical form that deep neural networks can understand.

Documentation and examples for common text utilities to control alignment, wrapping, weight, and more.

NounEdit. text (countable and uncountable, plural texts). A writing consisting of multiple glyphs, characters, symbols or sentences.

Fish are gill-bearing aquatic craniate animals that lack limbs with digits. 

I tried to determine the optimal number of clusters with elbow method, but unfortunately with no success. So I remove the stopwords and vectorised texts in the column:

import re
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
import nltk
from nltk.corpus import stopwords
from sklearn.pipeline import Pipeline

stop_words = stopwords.words('english')

def preprocessing(line):
    line = re.sub(r"[^a-zA-Z]", " ", line.lower())
    words = word_tokenize(line)
    words_lemmed = [WordNetLemmatizer().lemmatize(w) for w in words if w not in stop_words]
    return words_lemmed

  vect =TfidfVectorizer(tokenizer=preprocessing)
    kmeans =KMeans(n_clusters=3).fit(vect_text)

       n_clusters=3, n_init=10, n_jobs=1, precompute_distances='auto',

Is it right to cluster the texts above based on their similarity using this approach?


1 Answer 1


So, the problem is how to cluster texts.

Firstly, an alternative approach to representing document meaning, you can use Doc2Vec and compare similarities between document embeddings (https://medium.com/wisio/a-gentle-introduction-to-doc2vec-db3e8c0cce5e).

Secondly, if you are unsure about the ideal number of clusters, instead of using k-means, you can use agglomerative clustering, which is essentially bottom-up method which clusters individual document embeddings by a distance metric to eventually merge all of the clusters to a "mega-cluster", containing all documents.

You can then see how the documents cluster together using a dendrogram.

Here is a good resource on agglomerative clustering : https://stackabuse.com/hierarchical-clustering-with-python-and-scikit-learn/


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.