# Collapse a list to most common spellings

I have a user-generated list of proper names. For the sake of conversation, imagine it's pet names.

This list will have many variations of the same name: misspellings, alternative spellings, and even random use of punctuation or spaces.

eg

Fluffy
Fluffy the Dog
Flufy Dog
Fllufy


I can get a count of each spelling to figure out that, say, Fluffy is the most common at 2,000 occurrences while the others are all <100.

I tried to condense using the Levenshein distance package (python-Levenshtein) which had mixed success. I would assign a name to a "more likely" name if it's Levenshtein distance was < 3 and the "more likely" name had a higher count. When I manually search a few known common names, I can manually weed out the Levenshein winners from the losers, as well as include others Levenshein missed in my list (eg Fluffy the Dog and Fluffy have a huge distance of 8).

Are there better techniques I should be trying?

• You can extract and compare the n-grams to deal with spelling mistakes.
– Emre
Nov 2 '16 at 7:02

Credit to @Emre for setting me on this path with n-grams.

I discovered the library fuzzywuzzy which is well-built to solve this exact problem. I'll give a brief synopsis, but a well-written tutorial and documentation made this pretty straight-forward.

I have a DataFrame where names are the index and there is a count column.
I loop through the index, finding high-similarity strings and then mapping the name to the name (including self) with the highest count.

names = df.index

#iterate through list find the top 10 (arbitrary) matching strings > 90 score, pop off the highest-volume order one
def find_top_match(name):
my_matches = process.extract(name, names, limit=10, scorer=fuzz.token_set_ratio)
my_matches = [t[0] for t in my_matches if t[1] >= 90]
top_match = df.loc[my_matches].sort_values(by='count', ascending=False).index[0]
return(top_match)

top_matches_set = []

for c in df.index:
returned_match = find_top_match(c)
top_matches_set.append(returned_match)

df['real_name'] = top_matches_set