I have the following three sentences, extracted from a dataframe. I would like to check the similarity and create clusters based on their level of similarity.

Authors       Sentences
John Smith   Some people do not completely understand the risk of UV rays. 
Jane Lower   People do not understand the risk of UV rays, wrote the journalist in the Herald. 
Craig Avatan In Berlin, people do not know how dangerous can be for their health a long exposure to UV rays. 

I would need to cluster them based on words and their sequences (like plagiarism). I have tried to use k-means, but I have not completely understood how to create clusters to plot. Something like this:

enter image description here

I have tried to use k-mean as follows:

def sent_tokenization (line):
    line = re.sub(r"[^a-zA-Z]", " ", line.lower())
    words = word_tokenize(line)
    words_lemmed = [WordNetLemmatizer().lemmatize(w) for w in words if w not in stop_words_list]
    return words_lemmed

tfidf_vect = TfidfVectorizer(tokenizer= sent_tokenization)
tfidf = tfidf_vect.fit_transform(df['Sentences'])

kmeans = KMeans(n_clusters=2).fit(tfidf)

However I am not able to plot the results. What I am looking for is something that can be easily visualised. Specifically, I would need to plot in a scatter plot as in the example, which can show the name of authors based on their sentence similarity, like in plagiarism. I am trying to see which authors have written similar texts.

In my example, I should have the first two authors closers than the third one, as their sentences are very similar (in terms of words and structure).

Could you please give me advice on how to plot/cluster the above information? If you need more information, feel free to ask.

  • $\begingroup$ Do you only have 3 samples? $\endgroup$ – Adelson Araújo May 17 at 1:02
  • $\begingroup$ I have around 2000 texts but most of them are similar to those I wrote above. Probably I should fix a threshold (or something else) for similarity to compare them to others. What I have been thinking is to groups these, than one by one checked the others and try to add them to the closest cluster. Does it make sense? If you like I could add more texts $\endgroup$ – Math May 17 at 1:05
  • $\begingroup$ If you are looking for a 2D visualization on this, you can apply tf-idf, then use a decomposition technique (e.g. PCA, NMF, or even LDA/topic modelling if you want to go further) for reducing the tf output dimensionality. Then you can scatterplot it. To group these points, you can use kmeans again, setting in the new plot the kmeans.label of each sample. $\endgroup$ – Adelson Araújo May 17 at 1:10
  • $\begingroup$ Do you think it would be something doable to show as example the steps you mentioned using my three sentences above? $\endgroup$ – Math May 17 at 1:16

It can be made in many ways. My starting approach would be (1) to apply the tf-idf as in your code snippet, then (2) reducing the output matrix to a lower dimensionality (say 2D for seeing it in a scatterplot) with a decomposition method (or with topic modelling approach), and finally (3) apply a clustering algorithm to visualize the samples/documents of each cluster.

However, this may have many flaws, because I don't know if the decomposition results would make any sense and I'm not aware of how many clusters are good enough, but anyway it would be useful as an exploratory analysis.

The logic would be something like that: (didn't tested)

# 1
tfidf_vect = TfidfVectorizer(tokenizer= sent_tokenization)
tfidf = tfidf_vect.fit_transform(df['Sentences'])

# 2
decomp_method = NMF(n_components=2)
tf_2d = decomp_method.fit_transform(tfidf) # output shape: [N, 2]

# 3
kmeans = KMeans(n_clusters=2)
tf_kmeans_labels = kmeans.fit(tf_2d).labels_ # array

# visualization
data = pd.DataFrame(tf_2d, columns=['c1', 'c2'])
data['kmeans_labels'] = tf_kmeans_labels
scatterplot(x='c1', y='c2', hue='kmeans_labels',data=dat) 
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.