I have to work on 2 datasets where I have to find out the duplicates in addresses present in both the dataset.
I am a bit confused that which one of the Levenshtein distance or cosine similarity, I should use in order to find the similarities between the addresses of both the dataset.
I am new to this, So any Constructive Advice or Suggestion related to this one are welcome. Thanks in Advance!!!
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
This problem is called record linkage. There are several related questions which might help:
- About general techniques in record linkage
- About Cosine vs. Levenshtein
- About efficiency in record linkage: see here and here
Generally using a token-based measure (like Cosine-TFIDF) only is not sufficient, because it cannot capture spelling variants and typos. Character-based methods (like Levenshtein edit distance) are better at this but they do not handle token swapping for instance. Depending how precise you want to go there are hybrid methods which try to combine the advantages of both.
-
$\begingroup$ Thanks a lot! Will definitely have a look into it $\endgroup$ Commented Aug 4, 2021 at 6:37