I have a user-generated list of proper names. For the sake of conversation, imagine it's pet names.

This list will have many variations of the same name: misspellings, alternative spellings, and even random use of punctuation or spaces.


Fluffy the Dog
Flufy Dog

I can get a count of each spelling to figure out that, say, Fluffy is the most common at 2,000 occurrences while the others are all <100.

I tried to condense using the Levenshein distance package (python-Levenshtein) which had mixed success. I would assign a name to a "more likely" name if it's Levenshtein distance was < 3 and the "more likely" name had a higher count. When I manually search a few known common names, I can manually weed out the Levenshein winners from the losers, as well as include others Levenshein missed in my list (eg Fluffy the Dog and Fluffy have a huge distance of 8).

Are there better techniques I should be trying?

  • $\begingroup$ You can extract and compare the n-grams to deal with spelling mistakes. $\endgroup$
    – Emre
    Nov 2 '16 at 7:02

Credit to @Emre for setting me on this path with n-grams.

I discovered the library fuzzywuzzy which is well-built to solve this exact problem. I'll give a brief synopsis, but a well-written tutorial and documentation made this pretty straight-forward.

I have a DataFrame where names are the index and there is a count column.
I loop through the index, finding high-similarity strings and then mapping the name to the name (including self) with the highest count.

names = df.index

#iterate through list find the top 10 (arbitrary) matching strings > 90 score, pop off the highest-volume order one
def find_top_match(name):
    my_matches = process.extract(name, names, limit=10, scorer=fuzz.token_set_ratio)
    my_matches = [t[0] for t in my_matches if t[1] >= 90]
    top_match = df.loc[my_matches].sort_values(by='count', ascending=False).index[0]

top_matches_set = []

for c in df.index:
    returned_match = find_top_match(c)

df['real_name'] = top_matches_set

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.