I'm using NearestNeighbors to do name matching and at a certain point the results become misaligned. My standardized list of names is 100s of millions. My list of names coming in to be matched is considerably smaller but still could be in the 250k to 500k range. After a certain point it appears the index begins to shift by 1 or more.
nbrs = NearestNeighbors(n_neighbors=1, n_jobs=-1).fit(tfidf)
unique_org = set(names['VariationName'].values) # set used for increased performance
#matching query:
def getNearestN(query):
queryTFIDF_ = vectorizer.transform(query)
distances, indices = nbrs.kneighbors(queryTFIDF_)
return distances, indices
print('Getting nearest n...')
distances, indices = getNearestN(unique_org)
unique_org = list(unique_org) #need to convert back to a list
print('Finding matches...')
matches = []
for i,j in enumerate(indices):
temp = [round(distances[i][0],2), clean_org_names.values[j][0][0],unique_org[i]]
matches.append(temp)
print('Building data frame...')
matches = pd.DataFrame(matches, columns=['Match confidence (lower is better)','Matched name','Original name'])
print('Data frame built')
It appears that once my standardized list gets over 80k it begins to shift the results down.
The "messy name" of VITALI, ANGELO (has a comma)
VITALI, ANGELO
The standardized list of names may include these(no comma)
VITALI ANGELO
SENSABLE TECHNOLOGIES INC
After running it through the above matching the result below shows that VITALI, ANGELO is a near perfect match to SENSABLE TECNOLOGIES INC, because the index has shifted down one...I think.
0.00 SENSABLE TECHNOLOGIES INC VITALI, ANGELO
Is it possible that the size or number of records is exceeding that matrix limits and it somehow messes the indices?
unique_org
is a set. In Python, sets are not guaranteed to be ordered. Which means theindices
could be different for different runs. $\endgroup$ – Brian Spiering Apr 29 '20 at 20:13