1
$\begingroup$

I have an input data file which contains list of drug names.

I have more than 1000 unique drug names. However, the drug names has spelling mistakes and space character issues.

For ex: we have ISONIAZID 300MG TAB , ISONAZID300MG TAB, ISNIAZID 300MG

You can see how the above 3 terms are different in representation (due to spelling mistake) but actually indicate the same drug ISONIAZID 300MG TAB (which is the right spelling)

But the problem is there are several other drugs with such spelling mistakes and I am not sure how can I group all of them into one (meaning rename it with right spelling)? ex: All the 3 terms above should be renamed to ISONIAZID 300MG TAB (which is the right spelling)

I am posting it here to seek your opinion on is there any medical dictionary or automated approach which can take my raw csv file as input and output proper drug names?

Improve this question
$\endgroup$
2
  • $\begingroup$ Try this library. $\endgroup$
    – ashutosh singh
    Commented Apr 21, 2020 at 14:27
  • $\begingroup$ It contains multiple methods to correct spellings. 1. Contextual correction 2. Edit Distances 3. Phonetic Correction. It doesn't support the word split problem though. $\endgroup$
    – ashutosh singh
    Commented Apr 21, 2020 at 14:27

1 Answer 1

Reset to default
2
$\begingroup$

There are several general approaches to this but almost all of them compare the values to some baseline and make a decision whether the individual value is close enough.

You can compare the similarity of strings using different methods e.g. I often use the Levenshtein distance which basically measures how many characters you would have to change to convert word a into word b. By grouping all words with a low enough Levenshtein distance you already identify all words that should be the same.

It is even easier if you have a dictionary of "correct values" in which case you would compare each value to all entries in the dictionary and assign it the value with the lowest levenshtein distance.

Using some basic text analysis you could also identify the most common spelling mistakes in your data and eliminate them based on replacement rules.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi @Fnguyen, Can you share any tutorial or example on how this is implemented? $\endgroup$
    – The Great
    Commented Jun 12, 2020 at 3:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.