I have a column that contains all texts that I would like to cluster in order to find some patterns/similarity among each other.
Text Word2vec is a two-layer neural net that processes text by “vectorizing” words. Its input is a text corpus and its output is a set of vectors: feature vectors that represent words in that corpus. While Word2vec is not a deep neural network, it turns text into a numerical form that deep neural networks can understand. Documentation and examples for common text utilities to control alignment, wrapping, weight, and more. NounEdit. text (countable and uncountable, plural texts). A writing consisting of multiple glyphs, characters, symbols or sentences. Fish are gill-bearing aquatic craniate animals that lack limbs with digits.
I tried to determine the optimal number of clusters with elbow method, but unfortunately with no success. So I remove the stopwords and vectorised texts in the column:
import re from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.cluster import KMeans from nltk.tokenize import word_tokenize from nltk.stem import WordNetLemmatizer import nltk from nltk.corpus import stopwords from sklearn.pipeline import Pipeline stop_words = stopwords.words('english') def preprocessing(line): line = re.sub(r"[^a-zA-Z]", " ", line.lower()) words = word_tokenize(line) words_lemmed = [WordNetLemmatizer().lemmatize(w) for w in words if w not in stop_words] return words_lemmed vect =TfidfVectorizer(tokenizer=preprocessing) vect_text=vect.fit_transform(df['Text']) kmeans =KMeans(n_clusters=3).fit(vect_text) n_clusters=3, n_init=10, n_jobs=1, precompute_distances='auto',
Is it right to cluster the texts above based on their similarity using this approach?