I'm using NearestNeighbors to do name matching and at a certain point the results become misaligned. My standardized list of names is 100s of millions. My list of names coming in to be matched is considerably smaller but still could be in the 250k to 500k range. After a certain point it appears the index begins to shift by 1 or more.
nbrs = NearestNeighbors(n_neighbors=1, n_jobs=-1).fit(tfidf) unique_org = set(names['VariationName'].values) # set used for increased performance #matching query: def getNearestN(query): queryTFIDF_ = vectorizer.transform(query) distances, indices = nbrs.kneighbors(queryTFIDF_) return distances, indices print('Getting nearest n...') distances, indices = getNearestN(unique_org) unique_org = list(unique_org) #need to convert back to a list print('Finding matches...') matches =  for i,j in enumerate(indices): temp = [round(distances[i],2), clean_org_names.values[j],unique_org[i]] matches.append(temp) print('Building data frame...') matches = pd.DataFrame(matches, columns=['Match confidence (lower is better)','Matched name','Original name']) print('Data frame built')
It appears that once my standardized list gets over 80k it begins to shift the results down.
The "messy name" of VITALI, ANGELO (has a comma)
The standardized list of names may include these(no comma)
SENSABLE TECHNOLOGIES INC
After running it through the above matching the result below shows that VITALI, ANGELO is a near perfect match to SENSABLE TECNOLOGIES INC, because the index has shifted down one...I think.
0.00 SENSABLE TECHNOLOGIES INC VITALI, ANGELO
Is it possible that the size or number of records is exceeding that matrix limits and it somehow messes the indices?