1
$\begingroup$

I have two lists to compare. List 1 contains strings which represent the strings in List2. Sometimes, they are direct matches, some times, one word match, sometimes there are acronyms etc. An example is as below: 

   List1           |  List2
  ------------------------
   great hall town | Great Hall
   grt_town        | Great Hall      
   gHall           | Great Hall
   great town      | Great Hall
   mind great town | Great Hall

There are two main problems:

  1. List2 is very large ( millions of records, all short in length)

  2. List1 records are very unrelated sometimes. There are longer prefixes, suffixes which are not in List2. Some words in List2 are reduced to a letter.

I vaguely have some thoughts around clustering List2 and trying to find distance between centroids and the elements from List1. Do you think this as a starting point? Thanks

Improve this question
$\endgroup$
2
  • $\begingroup$ What are you trying to achieve? What problem are you trying to solve? You give us the background about some data you have but don't tell us what you're trying to accomplish, so it's not clear what your specific question is. I encourage you to edit the question to clarify. $\endgroup$
    – D.W.
    Commented Jun 9, 2017 at 18:10
  • $\begingroup$ Why do you think "great town" should match "great hall" and not, say, "great wall" or "great fall"? $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Jun 12, 2017 at 7:41

1 Answer 1

Reset to default
1
$\begingroup$

You could try this:

  1. Take distinct records from List2
  2. Clean List1 of any extraneous characters or symbols
  3. Use a fuzzy logic package like Python's fuzzywuzzy. It has partial matching options.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.