1
$\begingroup$

I have bunch of text which I want to segregate based on semantic similarity. Running through K-Means, I was able to divide the complete text into different clusters.

However, I still need to find which cluster contains most similar text. For e.g :-

Cluster 1 :-
'Hey there, how are you'.
'How are you dear'.
'I haven't seen you for past three days. How are you'.

Cluster 2 :-
'As per survey conducted, smoking kills more people than ...'.
'Avoid smoking in public places.'.
'People more likely to avoid smoking if they ...'.
'As per survey conducted, people most likely chose democrats'.

Cluster 1 should rank above Cluster 2 as it has more semantic similarity among text contained within.

$\endgroup$

2 Answers 2

1
$\begingroup$

Generally, library that you use for k-means clustering will report "within cluster sum-of-squares" value after running k-means procedure. A smaller value means the cluster is tightly bound (most messages are similar to each other). You can use this value to order the cluster IDs.

$\endgroup$
0
$\begingroup$

Assuming you have clustered them according to some word/characters vectors, your clustering algorithm probably produced a centroid point for each cluster.

Now, the next step depends on what you have defined as a 'higher ranking'. Do you see clusters having very dense distribution of text as having a good rank? Or do you want something else (e.g clusters with no outliers) This depends your own definition of (taken from your question):

'contains most similar text'

Now, the next step will be to calculate the average euclidean distance/cosine similarity/some kind of distance metric between each piece of text in the cluster. Assuming you are interested in clusters of text which are the most similar, then the cluster with the lowest average distance between centroid and text vectors will be the highest ranked cluster.

But wait! There are more subtleties involved.

Three important things to note:

  1. The distance metric you use is crucial for success. Please read up on different metrics such as euclidean distance, manhattan distance, cosine similarity etc. Each of them is beneficial in their own ways and it depends on what you define as text similarity. You must know what you want - how would you treat texts with different lengths? Do you only want to consider uncommon/rare words in ranking similarity? Different metrics can be used depending on how you answer these questions.

  2. You may want to feature engineer your texts beforehand. For example, you could end up with clusters grouping texts containing many 'a' and 'the' and 'I' (which are more common in narrative texts instead of, say, news articles). You could perhaps remove these words beforehand using some existing library and stemmers.

  3. To visualise the spread of the clusters, you can try to use PCA to reduce the word vector space to 2D. Clustering algorithms are notorious for failing edge cases (see No Free Lunch Theorem). If your clustering algorithm has already failed in the first place, there is no point trying to rank them.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.