# Sentence similarity

Every week I get a group of sentences (~1000) each of them may be similar. Example:

• metallica hard wired
• metallica hardwire
• metallica hardwired
• metallica hard wire
• hardwired metallica
• hardwire metallica
• hardwire

I'm using Cosine similarity to find common documents and group them. I have realized that similar docs:

• metallica hardwire and metallica hardwired

return ~0.5 similarity.

hardwired metallica and metallica hardwire

return ~0.433

Other docs with more words return higher values. (Im using cosine_similarity from sklearn.metrics.pairwise)

I iterate over each document and get the similarity among all docs, after that I extract the highest values. (cosine similarity > 0.55)

So far is working fine but there are cases in which I can't find similar sentences unless I reduce my coefficient, doing so it may associate other values to non-related items.

I want to know what is the best technique to group common sentences from a list of sentences. Not sure if that would be semantic similarity.

• Why dont you stem the sentence before using similarity? – Hima Varsha Oct 4 '16 at 5:26

Cosine is only good for long documents. example document and exampl docunemt have 0 cosine similarity. Similarly, hard wired and hardwired are completely dissimilar for cosine. Because it is based on counting the number of identical words.
If you want letter-based similarity, consider levenshtein. But you will likely need to go to something complex n-gram based to also detect wordA wordB and wordB wordA.