I'm trying to find a solution for a data quality problem - specifically, identifying which items in different data sets are used to represent the same things.
As an example, assume that we're a retailer and we buy out a couple of other retailers. In the process, we also get their systems and databases. This might lead to some overlap - different systems can represent the same items, customers, etc. in different ways, but with no single unique identifier.
What would the best approach be to determine which rows represent the same thing across data sets in order to come up with a single view of 'unique' entities?
I've done a machine learning course, and I understand the bare minimum. I believe that the solution to this problem requires a clustering algorithm, but what type? I may be dealing with a multitude of features in the data - dimensions, names, dates, contact details - and some of those would probably require a higher 'weight' in matching.
A: ABC Notebook, Large, Released: 2018-02-20, 150mm x 100mm.
B: Notebook (ABC), L, Date: 18/02/2018, 150mm x 100mm
I'd expect these to be treated as the same item.
A: Doe, Jane, DOB 1970-06-23, 123 ML Ave, F
B: John Doe, DOB 1971-04-33, 123 ML Avenue, M
C: J. Doe, Born '71
I'd expected B and C to be identified as the same person, but not A.
For this scenario, I don't expect to come up with something with a 100% accuracy, but I would like to be able to come up with a (narrow) list of possible matches that someone can check.
Can some please point me in the right direction? Any case studies that I should look at?