NLP problem : Choosing the optimal parser for extracting quantitative values from text

Question

I have a clinical NLP problem for which I would need some help to establish a proper framework.

I am trying to extract different elements from echocardiography reports. Those elements are both quantitative and qualitative.

For example, quantitative elements are in the form :

"LVEF : 40%", "LVIM : 2 mm/s" , "the tricuspid regurgitation is 4mm/s"

and qualitative elements are in the form :

"The ventricular function is depressed", "We noticed reduced diastolic function"

---

Ultimately my objective is to obtain the following table for one note :

ID LVEF LVIM TR Diastolic Dysfunction
1  40   2    4  Reduced

---

My current framework to do so is to use the Quanteda package and Spacyr.

1. Tokenization of the text

   This is pretty straight forward.

2. Parsing

   This is where I am a little bit hesitant. I believe that it is key for this project to capture the sequence of the word as retrieving what number is associated with which parameter would be difficult otherwise (i.e. for example in a bag-of-word scenario, it would be impossible to know if 4 refers to the value of LVEF or LVIM for example).

   I do not know what would be the best way to parse the text to retrieve that information.

3. Dictionary use

   A lot of the echocardiogram parameters are registered in different ways. For example, 'LVEF' is documented as 'Left Ventricular Function', 'Heart Function', 'Left Function'. I constructed a dictionary mapping all the variations possible for a given concept.

   At what point of the NLP pipeline should the dictionary be used and how does Quanteda work with custom dictionaries ?

Thank you all!

Answer 1

This is an NER problem. Rather than you splitting your sentence to words and finding the right word from dict, I suggest you use an NER (may be spacy NER mentioned by @jindrich).

This NER will point out right block of information from the text your sentences.

Once you get an Entity then you can parse its value. If it is quantitative then it is easy to parse (with simple preprocessing) and if it is qualitative then you may have to convert string to numbers like one to 1 . There are free libraries for that.