0
$\begingroup$

I have a set of unique sentences. For each sentence I calculate a semantic similarity score (between 0 to 1) with the remaining sentences as mentioned in the below example.

E.g.,

Dataset = {sen1, sen2, sen3, sen4,..., senN}

For sen1 I calculate pairwise semantic similarity scores as follows.

sen1 and sen2 = 0.3
sen1 and sen3 = 0.7
sen1 and sen4 = 0.9
...
...
...
sen1 and senN = 1.0

Likewise for all the sentences I calculate pairwise semantic similarity scores.

Since, I am getting a pairwise value, is it possible to cluster these sentences? Also what is the most appropriate clustering technique in my situation?

(I want to cluster sentences based on the similarity value I have and also I consider values above 0.5 as semantically similar sentences.)

$\endgroup$
1
$\begingroup$

There are several techniques that you could apply in order to cluster data if your input is a matrix of pairwise distances between elements. As usual, the best option depends on your specific data, so it is hard to answer to the question of what is the best one, but you could try any of the following ones:

  • The k-medoids algorithm is similar to the well-known k-means algorithm. After randomly choosing k of your sequences as initial cluster centers (initial medoids) and assigning each sequence to the closest medoid, you randomly reassign sequences to different clusters as long as the value of the cost function decreases.
  • Hierarchical clustering is another example of clustering algorithm whose input is a matrix of pairwise distances between sequences. In this case the output is a dendrogram.
  • Another option is to apply multidimensional scaling, a dimensionality reduction technique which input is a matrix of pairwise distances between sequences, to project your sequences into a 2D plane. Once you do that, you can apply any cluster algorithm you can think of, like for instance k-means.

As I said, there are many other options, but these ones are the simplest ones I can think of, and the ones I would start from.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks a lot. As you have mentioned my input is a matrix where the diagonal is 1. I am only considering the scores above 0.5 as sentences that are semantically similar. Hence, can you please tell me if I should make the values lower than 0.5 as zero before I perform the clustering? $\endgroup$ – Smith Aug 11 '17 at 13:27
  • 1
    $\begingroup$ I think that by doing that you would be loosing information. I do not see any adavantage of doing so. $\endgroup$ – Pablo Suau Aug 13 '17 at 7:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.