I have around 4000 customer records and 6000 user records and about 3000 customer records match leaving 1000 unmatched customers. I have created a fuzzy matching algorithm using Levenshtein and Hamming and added weights to certain properties, but I want to be able to match the remaining records without manually doing this. Ideally I want to implement an algorithm to take a customer and user and output match/no match. However, wouldn't I need to train with true negatives? Is there an algorithm that can train with just 1 label? Thanks
$\begingroup$
$\endgroup$
1
-
3$\begingroup$ Hello, rodrigo! Read about record linkage and one-class classification. I think your approach is sound for a first pass. $\endgroup$– EmreMar 30, 2018 at 22:55
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
You can obtain one negative example by taking one of the 3000 customer records and pairing it with any user record that is known not to match. In this way, you can obtain $3000$ positives and $3000 \times 5999$ negatives. You could then train a boolean classifier on this entire training set. This might work better than using one-class classification on just the positives.
Even better might be to use techniques for learning to rank. If $c$ is a customer record that is known to match a user record $u$, and $u'$ is any other user record (which $c$ doesn't match), then you want your classifier to rank the pair $(c,u)$ higher than $(c,u')$. In this way you can obtain $3000 \times 5999$ such ranking-pairs, and try to train a classifier to learn to rank, then use that to find the best match for each of the 1000 customer records.
-
$\begingroup$ Wouldn't the first method just train the classifier to reject every pair that has customer c as a customer? @D.W. $\endgroup$ Mar 30, 2018 at 23:13
-
$\begingroup$ @RodrigoEstrella, no, why would it do that? If you generate all possible negatives, you'll have 3999 negatives per customer. I've edited my answer to try to make it clearer what I am suggesting. $\endgroup$– D.W.Mar 30, 2018 at 23:20
-
$\begingroup$ I ended up using your first method to offset my customers by 1000 and creating 3000 true negatives. Thanks! $\endgroup$ Apr 2, 2018 at 23:33
-
$\begingroup$ @RodrigoEstrella, FYI, using only 3000 negatives instead of all 3000*5999 negatives might make this perform significantly worse. If you want to get by with fewer negatives, my suggestion would be to obtain negatives by choosing the best non-match: i.e., for each customer record $c$ with a known match to a user record $u$, look at all the non-matching user records and find the one $u'$ that is closest to matching $c$ (but still doesn't match it); then add $(c,u)$ as a positive and $(c,u')$ as a negative. $\endgroup$– D.W.Apr 2, 2018 at 23:56