I'm trying to compare a list of names (duplicated into a clean file and a messy file). I then compare the files against each other. My problem is that it returns only the top 1 result for each, which is itself (the identical record in each file). What I am trying to capture is the second result, which would be the closest match, not being itself.
names = pd.read_csv('C:/Temp/messynames.txt', sep='\t') org_names = names['VariationName'].unique() vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams) tf_idf_matrix = vectorizer.fit_transform(org_names) clean_org_names = pd.read_csv('C:/Temp/cleannames.txt', sep='\t') org_name_clean = clean_org_names['StandardName'].unique() vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams, lowercase=False) tfidf = vectorizer.fit_transform(org_name_clean) nbrs = NearestNeighbors(n_neighbors=3, n_jobs=-1).fit(tfidf) unique_org = set(names['VariationName'].values) def getNearestN(query): queryTFIDF_ = vectorizer.transform(query) distances, indices = nbrs.kneighbors(queryTFIDF_) return distances, indices distances, indices = getNearestN(unique_org) unique_org = list(unique_org) #need to convert back to a list matches =  for i,j in enumerate(indices): temp = [round(distances[i],2), clean_org_names.values[j],unique_org[i]] matches.append(temp) matches = pd.DataFrame(matches, columns=['Match confidence (lower is better)','Matched name','Original name']) matches.to_csv('C:/Temp/matchednames.txt', sep='\t', encoding='utf-8', index=False, quoting=3)
For a file with the following four names:
NOKIA NOKIAA NOKIA LMD NOKIA LTD
The results looks like this:
Match confidence Matched name Original name 0 0.0 NOKIA LMD NOKIA LMD 1 0.0 NOKIAA NOKIAA 2 0.0 NOKIA NOKIA 3 0.0 NOKIA LTD NOKIA LTD
I'm trying to get to something more like:
Match confidence Matched name Original name 0 0.1 NOKIA LTD NOKIA LMD 1 0.1 NOKIA NOKIAA