Every week I get a group of sentences (~1000) each of them may be similar. Example:

  • metallica hard wired
  • metallica hardwire
  • metallica hardwired
  • metallica hard wire
  • hardwired metallica
  • hardwire metallica
  • hardwire

I'm using Cosine similarity to find common documents and group them. I have realized that similar docs:

  • metallica hardwire and metallica hardwired

return ~0.5 similarity.

hardwired metallica and metallica hardwire

return ~0.433

Other docs with more words return higher values. (Im using cosine_similarity from sklearn.metrics.pairwise)

I iterate over each document and get the similarity among all docs, after that I extract the highest values. (cosine similarity > 0.55)

So far is working fine but there are cases in which I can't find similar sentences unless I reduce my coefficient, doing so it may associate other values to non-related items.

I want to know what is the best technique to group common sentences from a list of sentences. Not sure if that would be semantic similarity.

  • 1
    $\begingroup$ Why dont you stem the sentence before using similarity? $\endgroup$ – Hima Varsha Oct 4 '16 at 5:26

Cosine is only good for long documents. example document and exampl docunemt have 0 cosine similarity. Similarly, hard wired and hardwired are completely dissimilar for cosine. Because it is based on counting the number of identical words.

If you want letter-based similarity, consider levenshtein. But you will likely need to go to something complex n-gram based to also detect wordA wordB and wordB wordA.

| improve this answer | |
  • $\begingroup$ get_distance('hardwire metallica','metallica hardwire') gives me a total of 16. I can think of tokenize sentence, order it and then get the distance?, when you mention complex n-gram what exactly you are referring, can you add more details? Thanks $\endgroup$ – gogasca Oct 5 '16 at 19:48
  • $\begingroup$ I don't know if anyone experimented with that. ngrams are quite expensive. Levenshtein already does not scale well to large data... $\endgroup$ – Has QUIT--Anony-Mousse Oct 5 '16 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.