I have a DataFrame with two columns, let's call them column A and column B. Each column is identified with a unique ID and a description.
Usually, each idA is in a 1-1 relation with idB. Unfortunately, people can make mistake. In the dataSet I have, a few IdA can be in a 1-n relation with idB...
My question is, how can I find the most suitable idB for each duplicated idA?
So far I tried to clean the data, tokenize them, lemmatize them. And then for each word in the descriptionA, compute a score with each word in the descriptionB according to the frequency of their common apparition.
Basically this is the code:
for wordA in descriptionA: for wordB in descriptionB: self.data[wordA][wordB] += 1
for wordA in descriptionA: for wordB in descriptionB: score += self.data[wordA][wordB]
Is there any other algorithm I could try? Does making NLP neural networks make sense here? If yes, how can I get started with it?
PS: The dataset being filled by people all other the world, there is a little language consistency, but not that much (this is basically in spanish, english and french, but mainly in english)