I am new to NLP, I have a bunch of raw data that is not tagged at all of medical questions, I need to extract from them what are the health issues stated in those texts.

I was thinking I need to create two custom POS tags for NER:
-the location on the body
-the problem itself

So if someone asked 'my head hurts' it would understand that the location is the head and the problem is that it hurts, but if someone asked 'my skin is red around my abdomen' it would understand that the location is the abdomen and the problem is that the skin is red.

After I extract this data I need to recommend medical articles based on what that user asked.
I have some questions:
1.Am I on the right path?
2.How would you implement it?
3.Do I need a custom pos tag for the location and the health issue or can it be done easier? How would you extract those informations?
4.I guess I have to manually tag questions right?
5.What framework would you use?
6.To create a recommender system I need to extract the same informations from medical articles?
7.How would you create a recommender system?

As I said, I am new to NLP and I didn't decide on the framework yet but the questions are not in english, I have found however on github a WordNet clone and a Named Entity Corpus for my language, so please keep in mind when recommending frameworks.


A few comments:

  • The term "POS" (Part Of Speech) has the specific meaning of "grammatical category" so it's not really what you mean here, you should probably say "custom entities tags/categories/labels".
  • Your plan makes sense, but be aware that that you're addressing a really difficult problem so even with a really good dataset and method it's not going to work perfectly (probably far from it).
  • You're probably going to need a very large amount of annotated data for this to work, because there are many different ways to express a medical problem so the model will need many examples to extract the relevant patterns.
  • The standard way to address this kind of problem is a bit different when there are terminology resources for the language: the medical terms can be annotated directly simply by pattern matching. If you have such resources and they are complete enough, this saves you the need to manually annotate and train a custom model.
  • Doing a recommender system on top of that is a second problem, and I think you need to think carefully about the goal:
    • if it's meant to be used by non-experts, it wouldn't make sense to recommend research articles.
    • The vast majority of the research literature is published in English, so there would be an additional problem to address here.
    • In case you're considering the whole medical literature, you might have a complexity issue because it's huge (PubMed contains around 30 millions abstracts, and that's without considering the full articles in PMC)
    • What would be the basis for recommendation? Is it based only on matching the terms in the query with the documents?
  • $\begingroup$ Thanks for the response, the medical articles will be articles about certain problems not research papers, so if someone has back pain this is the type of article he will be recommended. I was thinking to extract the information from articles the same way and recommend it based on how common they are by applying cosine similarity between them $\endgroup$ – Aintsmartenough May 13 at 10:54
  • $\begingroup$ @Aintsmartenough ok, that makes sense. $\endgroup$ – Erwan May 13 at 12:21

Not the answer you're looking for? Browse other questions tagged or ask your own question.