0
$\begingroup$

We have a problem where we have a standardized database of Medicine names. On the other hand, there is a subset of medicine names which could have spelling mistakes, different structure or hypens, missing words etc. There is also some metadata available, like manufacturer name, unit size etc.

Human can easily map those two database with each other. We have used some string comparison and created some probabilistic scoring and in some cases it serves the purpose.

But lot of times we are running into lot of nuanced issues and conditions are keep getting piled up. Is there any idea if any machine learning type of algorithm can help? I have basic understanding of all major algorithms but yet I am drawing blank for this problem. Simple example is mapping Epilex 300mg tab with Epilex 300 Tablet. I can give more examples if needed.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ It seems that the bigger issue is data quality. Can you post some examples of names that are the same but are different strings? Without knowing much more, I'd say that fuzzy matching should do a decent job. But you should it to fix your database $\endgroup$
    – Valentin Calomme
    May 1, 2020 at 7:13
  • 1
    $\begingroup$ I do not think you need machine learning for this. If you have not yet done so: Try ngrams and then calculate a similarity score. More infos about ngrams: en.wikipedia.org/wiki/N-gram $\endgroup$
    – toom
    May 1, 2020 at 7:54

1 Answer 1

Reset to default
0
$\begingroup$

I think there is no need to use ML for this problem. We can solve this by a simple lookup table/map table. The thing is we have to update the lookup table whenever we encountered a new category.

To apply ML to this problem. You should have the data of a set of preprocessed, processed responses like [Epilex 300mg(preprocessed), Epilex 300(processed)]. If you have this data, you can apply the decision tree(without pruning) model to predict the processed text, given an unprocessed text. Remember you will get predictions with only the categories within the trained categories. What you can do is, if a new category has occurred you can manually add that response to train data. And train your data again. If next time that category is encountered it will predict correctly. Like this you can improve your model prediction power. After some time you can able to predict every response correctly. The screen shorts of sample code is attached here.

Sample Data

Solution

To have actual code consider this Link.

Improve this answer
$\endgroup$
2
  • $\begingroup$ But this seems like exactly a lookup table. the problem is there are so many variations of it, that eventually it become hard coded solution. So, I am trying to find a solution where algorithm adjusts itself. May be I will create a table of examples and post it here to make problem clear $\endgroup$
    – Kaushal Shah
    May 2, 2020 at 4:38
  • $\begingroup$ Sharing a complex example will always help. If the preprocessed text is really complex, the comments by toom, Valentin Calomme are helpful. And to get better results you can try ensembles methods, but if the cost of the misclassification is high, we need the human intervention at the end. $\endgroup$
    – Venkatesh Gandi
    May 2, 2020 at 5:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.