Every week I get a group of sentences (~1000) each of them may be similar. Example:
- metallica hard wired
- metallica hardwire
- metallica hardwired
- metallica hard wire
- hardwired metallica
- hardwire metallica
I'm using Cosine similarity to find common documents and group them. I have realized that similar docs:
- metallica hardwire and metallica hardwired
return ~0.5 similarity.
hardwired metallica and metallica hardwire
Other docs with more words return higher values. (Im using cosine_similarity from sklearn.metrics.pairwise)
I iterate over each document and get the similarity among all docs, after that I extract the highest values. (cosine similarity > 0.55)
So far is working fine but there are cases in which I can't find similar sentences unless I reduce my coefficient, doing so it may associate other values to non-related items.
I want to know what is the best technique to group common sentences from a list of sentences. Not sure if that would be semantic similarity.