Record Linkage problem

Question

I am building Matching Alogoritm using ML.Project is to match Internal customer data with external customer data.Features are names,address,city,state and zip.

We create pairs between data sets and calculate cosine similarity and then pass cosine values for all features pairs to Gaussian Mixture model.We started with 2 cluster, with expectation of one match cluster and one no match cluster.But ML does not build one match cluster and matches are in both the clusters.

Before passing to ML, i use Standard scaler and minmax scaler , but still don't get a clear nomatch and match cluster.If we increase the cluster same thing happens.

Match could be High cosine similarity in Name,Address,State,City & zip or Name ,address ,zip or any other combinations.We are dealing with huge volume , so we are using Spark ML.

How can we achieve optimal clustering?

Answer 1

I second Anony-Mousse's answer. There's a fair amount of literature on record linkage, it's worth spending some time exploring it to obtain decent results.

I assume that you use cosine on tokens? This is not sufficient, for persons names and addresses one needs to use measures based on characters (e.g. Levenshtein edit distance, Jaro and variants) or characters n-grams (e.g. cosine, Jaccard, etc.). You can search for string similarity metrics. For names you should preferarbly use a combination of both characters and token levels.

Now since you are trying unsupervised methods I assume that you don't have labeled data. In this case, a standard workaround is to use some form of bootstrapping:

1. using an simple unsupervised heuristic which scores pairs of records by their similarity, extract the top N pairs. This subset is likely to contain a large number of positives, whereas in the full data positives make up a very small proportion.
2. manually annotate the top N pairs: this is feasible as opposed to annotating the whole data, and that gives you a training set but not a representative one.
3. use a semi-supervised method (possibly active learning) to predict labels on the rest of the data, keeping in mind that the original training set is skewed towards positive. Typically it makes sense to assume a high probability of negative for the rest of the data.

Answer 2

Clustering is the wrong tool for this. And so is cosine similarity, bag of words model etc.

Obviously, standard scaling on house numbers and zip codes is a svery bad idea, don't you think?

There are special techniques developed for this kind of data with a reason. For example, it's unlikely that the customer entered house number 100 but lives in 5. That single difference is a very clear indication that it probably is not the same customer. On the other hand, "Park Rd." And "Parkrd." almost certainly is the same, but in cosine similarity that is three tokens difference, pretty big.

Clustering is an explorative technique, not one to achieve automatic matching. Choose other tools instead.