sklearn NearestNeighbors returning misaligned results

Question

I'm using NearestNeighbors to do name matching and at a certain point the results become misaligned. My standardized list of names is 100s of millions. My list of names coming in to be matched is considerably smaller but still could be in the 250k to 500k range. After a certain point it appears the index begins to shift by 1 or more.

nbrs = NearestNeighbors(n_neighbors=1, n_jobs=-1).fit(tfidf) 
unique_org = set(names['VariationName'].values) # set used for increased performance
#matching query:
def getNearestN(query):
  queryTFIDF_ = vectorizer.transform(query)
  distances, indices = nbrs.kneighbors(queryTFIDF_)
  return distances, indices

print('Getting nearest n...')
distances, indices = getNearestN(unique_org)

unique_org = list(unique_org) #need to convert back to a list
print('Finding matches...')
matches = []
for i,j in enumerate(indices):
  temp = [round(distances[i][0],2), clean_org_names.values[j][0][0],unique_org[i]]
  matches.append(temp)

print('Building data frame...')  
matches = pd.DataFrame(matches, columns=['Match confidence (lower is better)','Matched name','Original name'])
print('Data frame built') 

It appears that once my standardized list gets over 80k it begins to shift the results down.

The "messy name" of VITALI, ANGELO (has a comma)

VITALI, ANGELO

The standardized list of names may include these(no comma)

VITALI ANGELO
SENSABLE TECHNOLOGIES INC

After running it through the above matching the result below shows that VITALI, ANGELO is a near perfect match to SENSABLE TECNOLOGIES INC, because the index has shifted down one...I think.

0.00 SENSABLE TECHNOLOGIES INC VITALI, ANGELO

Is it possible that the size or number of records is exceeding that matrix limits and it somehow messes the indices?

Answer 1

You can refactor your code to make this issue easier to investigate.

Something like this:

nbrs = NearestNeighbors(n_neighbors=1, n_jobs=-1).fit(tfidf) 
orgs = list(set(names['VariationName'].values))

queryTFIDF_ = vectorizer.transform(orgs)
distances, indices = nbrs.kneighbors(queryTFIDF_)

matches = pd.DataFrame(columns=['Match confidence (lower is better)', 'Matched name', 'Original name'])

for distance, index, org in zip(distances, indices, orgs):
    match_confidence = round(distance, 2)
    matched_name = clean_org_names.values[index][0][0]
    matches.append(match_confidence, matched_name, org)