3
$\begingroup$

I need to extract product names from a text column in a dataset. Currently I'm using regular expressions to extract the product names from the middle of the text, but sometimes the product name is misspelled, incomplete or even amended in another word, which means that I am unable to identify and extract the product name.

We currently have around 1500-2000 products on that list and I have a data set with those products already identified from approximately 30,000 lines. Is there an approach that I can use this historical data to improve the identification of products that have not yet been identified?

Just an example:

The product X produced by the Company Y is used to treat skin diseases

Note: The product names doesn't appear in a fixed position.

Improve this question
$\endgroup$
1
  • $\begingroup$ String similarity measures would be an option, but it would probably require too much computation time to compare every subsequence of a sentence against every candidate name. A possible idea in this somewhat related question. $\endgroup$
    – Erwan
    Dec 19, 2020 at 0:27

1 Answer 1

Reset to default
3
$\begingroup$

What you need is perhaps Named Entity Recognition with custom entity dictionary. See this example:

enter image description here

Many packages like NLTK or Spacy have a large dictionary of such entities that enable models to identify them without using regular expressions. However, those pre-defined entities often is not applicable to one's application, thus won't recognize what you are looking for. That is why you need to spend time to update/add a list of such entities (products in your case). Also you mentioned that there are misspelling, variations etc. that make it harder. Here is where you need to expand your dictionary for such products with a fuzzy-matching algorithms to account for such edge cases programmatically.

You may start reading this article to the idea. More about Named Entity Recognition: a Blogpost, or the official Named Entity Recognition 101 by Spacy. Spacy supports adding products name in the Entity list, see page. Check out this question in stackoverflow too. But it is not easily available as you may have thought!!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.