I'm trying to compare a list of names (duplicated into a clean file and a messy file). I then compare the files against each other. My problem is that it returns only the top 1 result for each, which is itself (the identical record in each file). What I am trying to capture is the second result, which would be the closest match, not being itself.
names = pd.read_csv('C:/Temp/messynames.txt', sep='\t')
org_names = names['VariationName'].unique()
vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams)
tf_idf_matrix = vectorizer.fit_transform(org_names)
clean_org_names = pd.read_csv('C:/Temp/cleannames.txt', sep='\t')
org_name_clean = clean_org_names['StandardName'].unique()
vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams, lowercase=False)
tfidf = vectorizer.fit_transform(org_name_clean)
nbrs = NearestNeighbors(n_neighbors=3, n_jobs=-1).fit(tfidf)
unique_org = set(names['VariationName'].values)
def getNearestN(query):
queryTFIDF_ = vectorizer.transform(query)
distances, indices = nbrs.kneighbors(queryTFIDF_)
return distances, indices
distances, indices = getNearestN(unique_org)
unique_org = list(unique_org) #need to convert back to a list
matches = []
for i,j in enumerate(indices):
temp = [round(distances[i][0],2), clean_org_names.values[j][0][0],unique_org[i]]
matches.append(temp)
matches = pd.DataFrame(matches, columns=['Match confidence (lower is better)','Matched name','Original name'])
matches.to_csv('C:/Temp/matchednames.txt', sep='\t', encoding='utf-8', index=False, quoting=3)
For a file with the following four names:
NOKIA
NOKIAA
NOKIA LMD
NOKIA LTD
The results looks like this:
Match confidence Matched name Original name
0 0.0 NOKIA LMD NOKIA LMD
1 0.0 NOKIAA NOKIAA
2 0.0 NOKIA NOKIA
3 0.0 NOKIA LTD NOKIA LTD
I'm trying to get to something more like:
Match confidence Matched name Original name
0 0.1 NOKIA LTD NOKIA LMD
1 0.1 NOKIA NOKIAA