I am working on a project that involves dealing with manually entered text data. I have a dataset of customs records where the customs officers manually enter the name and address of companies importing goods from a certain country. Here is an example of the records:
| ID | buyer_string | | ------- | ------------------------------------------------- | | 5278898 | IVY LEAGUE (UK) LIMITED,THORNFIELD | | 5480749 | EI CORTE INGLES S.A. | | 206882 | NEXT PLCDESFORD RPAD ENDERBY LEICESTER LE19 4A... | | 4698965 | H&M HENNES & MAURITZ GBC AB SWEDEN N/P-H&M HEN... | | 3012744 | LOGIX FZCOPO BOX 261422 JEBEL ALI FREE ZONE UN... | | 5557832 | K-MART CORPORATION 3333 BEVERLY ROAD HOFFMAN E... | | 3000435 | TAHA KONFEKSIYON SANAYI TIC. LTD | | 61883 | H & M (INTL) LTD SUITE 2102B | | 3994401 | H&M HENNES & MAURITZ GBC AB | | 2767566 | WAL-MART CANADA CORP |
The task is to map each entry in the dataset to the name of the company it refers to. For instance:
- NEXT PLCDESFORD RPAD ENDERBY LEICESTER LE19 4A... -> Next PLC
- H&M HENNES & MAURITZ GBC AB SWEDEN N/P-H&M HEN... -> H&M Hennes and Mauritz
- WAL-MART CANADA CORP -> Walmart Corporation
The main problem is that I do not have a complete "clean" list of company names that I know are present in the dataset.
I have a small list of the biggest 200 companies operating in the country in question. Performing fuzzy matching using this list only covers around 50% of the records that I have (there are approximately 6 million records in total).
I have also tried clustering the records using Affinity Propagation (with the Levenshtein/Edit Distance as a distance metric). The clustering approach managed to cover around 80% of the records; however, many of cluster exemplars do not map onto "clean" company names.
My current pipeline is as follows:
- Normalize the text in each record i.e. convert everything to lowercase, remove special characters, etc.
- Using regular expressions, match records with the list of 200 company names.
- Perform affinity propagation clustering on the remaining unmatched records; using the first 30 characters as features.
- Perform fuzzy matching on the cluster exemplars to match them to actual company names (from the list that I have).
The issue with my approach is that I am unable to "discover" new companies in the dataset. For example:
- IVY LEAGUE (UK) is not in list of companies that I have, so there is no way I can map IVY LEAGUE (UK) LIMITED,THORNFIELD to IVY LEAGUE (UK).
So I was wondering if can get any other suggestions on tackling this task or at least improving my current pipeline?
Any help on this is much appreciated.