0
$\begingroup$

I'm working with textual data from medical field. I have a list of words and I want to build an algorithm that can classify each word into one or more given categories, like

  • Medicine_Name
  • Disease_Name
  • Body_Part

etc.

Please give me some ideas to implement it, specifically how to prepare the training example and implement the training phase.

$\endgroup$
2
  • 1
    $\begingroup$ I'm voting to close this question as off-topic because it shows a lack of research! $\endgroup$
    – Dawny33
    Jan 30 '16 at 12:07
  • 2
    $\begingroup$ I'm not sure "lack of research" needs to be as much a thing here as on Stack Overflow (maybe a good discussion topic for Data Science meta?). However, I also think there is a problem. The question is too broad, someone trying to answer it would need to write a relatively long review of text processing in general. The question also does not give enough information about the data ("textual data" could many things). Please add more description covering the starting data - e.g. is there labeled data to use supervised learning with, does the text appear in any context, or is it purely a word list? $\endgroup$ Jan 30 '16 at 16:37
2
$\begingroup$

The task you are asking about is Named entity recognition

A good way to identify a word category is by using patterns for the context of the word.

In your case you can use patterns like:

  • "I took X" for a medicine name
  • "I had pain in my X" for a body part
  • "I suffered from X" for a disease

The easiest and fastest way is to discuss the pattern with a domain expert and do a sanity check on the data you have.

A more systematic approach is to deduce the pattern from your data. I guess you can get a list of samples of the items. The technical details depend on the patterns you use but for this example let consider tokenization. Take the context of k-words next to the items on the list. Compare it to the context of other words. Contexts that are related to your items are suitable as patterns.

Bare in mind that these pattern will have very different precision and recall and they will lead to errors. It is important that you will have a labeled dataset so you will be able to estimate the patterns performance.

It will be fairly easy to get pretty good classification. It might be very hard to get a very good classification.

$\endgroup$
1

Not the answer you're looking for? Browse other questions tagged or ask your own question.