I'm using NearestNeighbors to do name matching and at a certain point the results become misaligned. My standardized list of names is 100s of millions. My list of names coming in to be matched is considerably smaller but still could be in the 250k to 500k range. After a certain point it appears the index begins to shift by 1 or more.

nbrs = NearestNeighbors(n_neighbors=1, n_jobs=-1).fit(tfidf) 
unique_org = set(names['VariationName'].values) # set used for increased performance
#matching query:
def getNearestN(query):
  queryTFIDF_ = vectorizer.transform(query)
  distances, indices = nbrs.kneighbors(queryTFIDF_)
  return distances, indices

print('Getting nearest n...')
distances, indices = getNearestN(unique_org)

unique_org = list(unique_org) #need to convert back to a list
print('Finding matches...')
matches = []
for i,j in enumerate(indices):
  temp = [round(distances[i][0],2), clean_org_names.values[j][0][0],unique_org[i]]

print('Building data frame...')  
matches = pd.DataFrame(matches, columns=['Match confidence (lower is better)','Matched name','Original name'])
print('Data frame built') 

It appears that once my standardized list gets over 80k it begins to shift the results down.

The "messy name" of VITALI, ANGELO (has a comma)


The standardized list of names may include these(no comma)


After running it through the above matching the result below shows that VITALI, ANGELO is a near perfect match to SENSABLE TECNOLOGIES INC, because the index has shifted down one...I think.


Is it possible that the size or number of records is exceeding that matrix limits and it somehow messes the indices?

  • $\begingroup$ When limiting my "name" column to only 50 characters the limit raises from 80k to roughly 250k in the standardized list before it starts having miss aligned results. It seems like there is a limit somewhere I can't nail down. Either server memory or the python variables being used. Still searching. $\endgroup$ – BamBamBeano Apr 28 '20 at 10:47
  • $\begingroup$ unique_org is a set. In Python, sets are not guaranteed to be ordered. Which means the indices could be different for different runs. $\endgroup$ – Brian Spiering Apr 29 '20 at 20:13
  • $\begingroup$ Have you tried calculating the levenshtein distance between names and then selecting the right threshold. pypi.org/project/python-Levenshtein $\endgroup$ – Carlos Mougan Apr 30 '20 at 8:07

You can refactor your code to make this issue easier to investigate.

Something like this:

nbrs = NearestNeighbors(n_neighbors=1, n_jobs=-1).fit(tfidf) 
orgs = list(set(names['VariationName'].values))

queryTFIDF_ = vectorizer.transform(orgs)
distances, indices = nbrs.kneighbors(queryTFIDF_)

matches = pd.DataFrame(columns=['Match confidence (lower is better)', 'Matched name', 'Original name'])

for distance, index, org in zip(distances, indices, orgs):
    match_confidence = round(distance, 2)
    matched_name = clean_org_names.values[index][0][0]
    matches.append(match_confidence, matched_name, org)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.