-1
$\begingroup$

I am working on a project that involves dealing with manually entered text data. I have a dataset of customs records where the customs officers manually enter the name and address of companies importing goods from a certain country. Here is an example of the records:

| ID      | buyer_string                                      |
| ------- | ------------------------------------------------- |
| 5278898 | IVY LEAGUE (UK) LIMITED,THORNFIELD                |
| 5480749 | EI CORTE INGLES S.A.                              |
| 206882  | NEXT PLCDESFORD RPAD ENDERBY LEICESTER LE19 4A... |
| 4698965 | H&M HENNES & MAURITZ GBC AB SWEDEN N/P-H&M HEN... |
| 3012744 | LOGIX FZCOPO BOX 261422 JEBEL ALI FREE ZONE UN... |
| 5557832 | K-MART CORPORATION 3333 BEVERLY ROAD HOFFMAN E... |
| 3000435 | TAHA KONFEKSIYON SANAYI TIC. LTD                  |
| 61883   | H & M (INTL) LTD SUITE 2102B                      |
| 3994401 | H&M HENNES & MAURITZ GBC AB                       |
| 2767566 | WAL-MART CANADA CORP                              |

The task is to map each entry in the dataset to the name of the company it refers to. For instance:

  • NEXT PLCDESFORD RPAD ENDERBY LEICESTER LE19 4A... -> Next PLC
  • H&M HENNES & MAURITZ GBC AB SWEDEN N/P-H&M HEN... -> H&M Hennes and Mauritz
  • WAL-MART CANADA CORP -> Walmart Corporation
  • etc...

The main problem is that I do not have a complete "clean" list of company names that I know are present in the dataset.

I have a small list of the biggest 200 companies operating in the country in question. Performing fuzzy matching using this list only covers around 50% of the records that I have (there are approximately 6 million records in total).

I have also tried clustering the records using Affinity Propagation (with the Levenshtein/Edit Distance as a distance metric). The clustering approach managed to cover around 80% of the records; however, many of cluster exemplars do not map onto "clean" company names.

My current pipeline is as follows:

  1. Normalize the text in each record i.e. convert everything to lowercase, remove special characters, etc.
  2. Using regular expressions, match records with the list of 200 company names.
  3. Perform affinity propagation clustering on the remaining unmatched records; using the first 30 characters as features.
  4. Perform fuzzy matching on the cluster exemplars to match them to actual company names (from the list that I have).

The issue with my approach is that I am unable to "discover" new companies in the dataset. For example:

  • IVY LEAGUE (UK) is not in list of companies that I have, so there is no way I can map IVY LEAGUE (UK) LIMITED,THORNFIELD to IVY LEAGUE (UK).

So I was wondering if can get any other suggestions on tackling this task or at least improving my current pipeline?

Any help on this is much appreciated.

$\endgroup$
0
$\begingroup$

The main problem is that I do not have a complete "clean" list of company names that I know are present in the dataset.

You have a garbage in, garbage out or business process problem. I would suggest:

  1. When users enter company information, they should be prompted first to select from a clean list and only type in a value if they say it is not present.
  2. After the name of the organization, enter some unique ID at the end in []. Hopefully the imported goods have that on the package. Then parse that unique number out and use that matching.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.