I have a DataFrame with two columns, let's call them column A and column B. Each column is identified with a unique ID and a description.

Usually, each idA is in a 1-1 relation with idB. Unfortunately, people can make mistake. In the dataSet I have, a few IdA can be in a 1-n relation with idB...

My question is, how can I find the most suitable idB for each duplicated idA?

So far I tried to clean the data, tokenize them, lemmatize them. And then for each word in the descriptionA, compute a score with each word in the descriptionB according to the frequency of their common apparition.

Basically this is the code:

training :

for wordA in descriptionA:
    for wordB in descriptionB:
        self.data[wordA][wordB] += 1


for wordA in descriptionA:
    for wordB in descriptionB:
        score += self.data[wordA][wordB]

Is there any other algorithm I could try? Does making NLP neural networks make sense here? If yes, how can I get started with it?

PS: The dataset being filled by people all other the world, there is a little language consistency, but not that much (this is basically in spanish, english and french, but mainly in english)


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.