I need to find similarities in newly added data using machine learning.
We have a huge data set with more than 200 000 rows having following information:
- City (From predefined list)
- Phone number (Predefined number format)
And many others.
The idea is that lots of people are registering each day for multiple services provided by our NGO.
What we really want is when a patient/user is coming to register, to check if similar data already exists in our database, so data is not duplicated, which will help us later to generate a unique list of beneficiaries.
If a user already exists with the following info:
Phone number: 123456789
And after a few months, he came to apply for another service without having his ID card, and while data entry errors do exist in our field of work and name could be written in different ways, the beneficiary will ask for a new registration, so instead of having 1 beneficiary registered in 2 services, we will have 2 beneficiaries with 2 different enrolled services. Which is not good for donors reporting:
Name: ABc (Assuming here ABC is different than ABc in a specific language)
Phone number: 1234567894
I need to run the new data into a model that will return a percentage of similarity, which means:
These data are 95% similar to already existed data (training set)
Any idea on how to do it? Which machine learning techniques should be used assuming that we have programmers who knows python and the basics of ML?