I need to find similarities in newly added data using machine learning.

We have a huge data set with more than 200 000 rows having following information:

  1. Name
  2. D.O.B
  3. City (From predefined list)
  4. Phone number (Predefined number format)

And many others.

The idea is that lots of people are registering each day for multiple services provided by our NGO.

What we really want is when a patient/user is coming to register, to check if similar data already exists in our database, so data is not duplicated, which will help us later to generate a unique list of beneficiaries.

If a user already exists with the following info:

Name: ABC

D.O.B: 1990-10-01

City: Lome

Phone number: 123456789

Service: Medications

And after a few months, he came to apply for another service without having his ID card, and while data entry errors do exist in our field of work and name could be written in different ways, the beneficiary will ask for a new registration, so instead of having 1 beneficiary registered in 2 services, we will have 2 beneficiaries with 2 different enrolled services. Which is not good for donors reporting:

Name: ABc (Assuming here ABC is different than ABc in a specific language)

D.O.B: 1990-10-01

City: Lomeh

Phone number: 1234567894

Service: Shelter

I need to run the new data into a model that will return a percentage of similarity, which means:

These data are 95% similar to already existed data (training set)

Any idea on how to do it? Which machine learning techniques should be used assuming that we have programmers who knows python and the basics of ML?

  • $\begingroup$ Are you going to do it at each registration or at some periods like monthly, daily etc? $\endgroup$ – Ilker Kurtulus Nov 21 at 12:54
  • $\begingroup$ @ilker its daily registration $\endgroup$ – alim1990 Nov 24 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.