2
$\begingroup$

I'm trying to compare a list of names (duplicated into a clean file and a messy file). I then compare the files against each other. My problem is that it returns only the top 1 result for each, which is itself (the identical record in each file). What I am trying to capture is the second result, which would be the closest match, not being itself.

names =  pd.read_csv('C:/Temp/messynames.txt', sep='\t')
org_names = names['VariationName'].unique()
vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams)
tf_idf_matrix = vectorizer.fit_transform(org_names)

clean_org_names = pd.read_csv('C:/Temp/cleannames.txt', sep='\t')
org_name_clean = clean_org_names['StandardName'].unique()
vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams, lowercase=False)
tfidf = vectorizer.fit_transform(org_name_clean)

nbrs = NearestNeighbors(n_neighbors=3, n_jobs=-1).fit(tfidf) 
unique_org = set(names['VariationName'].values)

def getNearestN(query):
  queryTFIDF_ = vectorizer.transform(query)
  distances, indices = nbrs.kneighbors(queryTFIDF_)
  return distances, indices
distances, indices = getNearestN(unique_org)

unique_org = list(unique_org) #need to convert back to a list

matches = []
for i,j in enumerate(indices):
  temp = [round(distances[i][0],2), clean_org_names.values[j][0][0],unique_org[i]]
  matches.append(temp)

matches = pd.DataFrame(matches, columns=['Match confidence (lower is better)','Matched name','Original name'])
matches.to_csv('C:/Temp/matchednames.txt', sep='\t', encoding='utf-8', index=False, quoting=3)

For a file with the following four names:

NOKIA
NOKIAA
NOKIA LMD
NOKIA LTD

The results looks like this:

Match confidence                         Matched name Original name
0                                 0.0    NOKIA LMD     NOKIA LMD
1                                 0.0       NOKIAA        NOKIAA
2                                 0.0        NOKIA         NOKIA
3                                 0.0    NOKIA LTD     NOKIA LTD

I'm trying to get to something more like:

Match confidence                         Matched name Original name
0                                 0.1    NOKIA LTD     NOKIA LMD
1                                 0.1        NOKIA        NOKIAA
$\endgroup$

1 Answer 1

2
$\begingroup$

My guess would be that in the following piece of code:

for i,j in enumerate(indices):
  temp = [round(distances[i][0],2), clean_org_names.values[j][0][0],unique_org[i]]
  matches.append(temp)

The variable (array element):

distances[i][0]

contains the top element match for the $i^{th}$ line. Replace the second array index so it becomes:

distances[i][1]

You may want to increase one of the two zero indices in the following line as well to have a distance like value:

clean_org_names.values[j][0][0]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.