2
$\begingroup$

Given: Bella Pharma Rosuvas 5 Enalapril 10 Domperidone 10 Ned's 24 by 7 PCM 650 Teneligliptin 5 ...

Get: (1,2,shop), (3,4,med), (5,6,med), (7,8,med), (9,12,shop), (13,14,med), ... i.e. words 1 and 2 denote a shop name, 3 and 4 denote a medicine name, etc.

Challenges:

  1. recognize shop names: some shop names maybe non-English dictionary words (Romanized spelling of local language words)
  2. separate medicine names: there are absolutely no punctuation marks like . or , and the pattern is like SmmmmSmmSmSmmmmmmmmSmm... (S is shop, m is full name of one medicine e.g. Rosuvas 5 or Amoxicillin Clavulanate)

Text is non-grammatical (just transcript of dictation between 2 people verifying inventory)

spaCy en_core_web_lg and other pre-build NER don't work (probably because both shop names and medicine names appear like proper nouns)

Also, there is no exhaustive list of medicine names: same chemical is sold under different names by different brands, sometimes weight/power specification may not be given.

Worth noting that chemical names have either suffix like *azole, *nate, *ide, *ril, etc. or weights like 650mg, 10mcg, etc. often (but not always) associated with them.

I am amazed how humans (even non-Pharmacists) can label most of the data correctly, how can I label this data almost as well as humans using ML/DL libraries?

I work as junior SDE, I can code just fine (e.g. convert BIO tag to JSON style spacy input) but have never used ML libraries before. Kindly include sample code in answer or link to same.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

NER requires indications in the context of the text to detect entities, so it's not surprising that it doesn't work here.

I am amazed how humans (even non-Pharmacists) can label most of the data correctly

The key certainly lies here: you should try to find out how humans do it, which indications they use, so that you can encode theses indications as features in order to predict the category.

My guess would be that you can rely on the pattern and identify only shop names: once the shop names are found, the rest are meds.

Improve this answer
$\endgroup$
0
$\begingroup$

A machine learning model can be trained on labeled data to learn patterns and memic how humans label the entities, however, the provided dataset plays an important role in enhancing the model performance.

The mentioned problem above can be easily transformed to a multi-label classification task and a BERT based model may perform very well in tagging those words in the text.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Could you provide a link or describe how the problem "can be easily transformed into a multi-label classification task"? $\endgroup$
    – Valentas
    Dec 1, 2022 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.