2
$\begingroup$

I have been provided with a text cleaning task and I am assuming this involves some amount of natural language processing.

I have a collection of names which does not have any specific pattern and I need to separate individual names from that list.

Below is an example:

John Batista Barack Obama George W. Bush Serafim Batzoglou Vijay Pande Fei Fei Li Oussama Khatib Kwame Nkrumah John Oliver

As you can see there are two things peculiar with the paragraph.

1) space " " is the only delimiter, however, it is unusable as there are names which have middle names.

2) I have french, German, Spanish, Chinese and other ethnic sounding names written in English.

Now the challenge I am facing is to find names from this paragraph. I am very new to NLP and use Python and R for modeling work.

Any help will be appreciated.

$\endgroup$

1 Answer 1

1
$\begingroup$

I'm not an expert here, so here's my (brute force?) method.

SeatGeek has open-sourced a python library called fuzzywuzzy which is great at text matching. It has a function called token_set_ratio that compares two multi-word strings and scores their distance. It can consider just the intersection of individual words and score only that intersection. eg "Barack Obama Bill Clinton Madeleine Albright" will get a score of 100 with "Bill Clinton", but also with "Bill Barack".

You'll need to build a list of known names, judging by the list they're famous people, which a quick google search shows hundreds of sites that have that ready for you. With that list, you can just loop your way through, computing a score with token_set_ratio. If the famous name beats some threshold score against your string, you append it to a list of captured names.

$\endgroup$
2
  • $\begingroup$ Alternatively, if your list is literally just [First Name] [Middle Initial (optional)] [Last Name], you could just string split on " ", throw out all elements with a regex [A-Z]\., and then take pairs. $\endgroup$
    – ScottieB
    Nov 29, 2016 at 18:21
  • $\begingroup$ Did you suggest using fuzzylogic? $\endgroup$ Nov 30, 2016 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.