I'm implementing a tool which should extract values of interest from unstructured text entries. The data set is several hundred thousands of medical entries. Each entry is relatively short (around 100 characters).
Each text entry contains a keyword MI (or TI or AO) and their repective value. The value is sometimes numeric and sometimes expressed as human language.
Some entries are very obvious. Some entries are more complex or ambiguous, in which case the end user of the tool (a doctor) should be able to make my tool "learn" a correct value.
Here are some examples with my comments
| Data entries | Expected extracted value | Comments |
|---|---|---|
| the value of MI is 1 | 1.0 | |
| MI = 1 | 1.0 | |
| MI > 1 | 1.5 | |
| MI < 1 | 0.5 | |
| No traces of MI found | 0.0 | |
| No traces of MI found | 0.0 | |
| traces of MI/TI found | 0.5 | |
| MI/TI/AO up to 4 | 3.5 | |
| MI/TI/AO up to 4 | 3.5 | |
| MI 2-[3] | 2.5 | |
| MI probably 2. Gradient of MI 170torr | 2 | This example shows that a numeric value following a MI does not always represent its value. Here 170 is not the value I'm looking for. It's the 2 |
| DOMV bez MS/MI v chlopni, ale úzký MI | ?? | Here I actually don't know what the value is. Only the user of the tool will be able to say what the value is in this case |
Those are just some non exhaustive examples. On important aspect is that the patterns repeat often across the data set.
Another important requirement is that what we are looking must be configurable. Sometimes we look for MI sometimes for TI.
I am myself a software engineer (not a data scientist nor a doctor) so I started with a naive implementation based on regular expressions. It's basically a set of regexp matched against each entry of the data set
This kind of works and I'm able to find around 90% of the cases. However it has the following drawbacks:
- the more regexp, the longer it will take.
- some regexp might clash and report different values for the same entry
- it might be cumbersome for a non tech user to add or adjust the regexps
The final user of the tool is not a software engineer but a doctor and does not know anything about regexp. However, in ambiguous/unclear cases it's the end user who can tell what the value is.
I could of course write a user interface to add/adjust/remove the rules (regexp) but that could be cumbersome as well
I was wondering if there is way without using explicit if/else rules but instead train a model which would be then used for finding the values.
I don't know much about NLP/AI or ML so maybe someone could point me to a right direction. Such as
- Would NLP or NER be the concept to use here?
- What could be some online services or open source tools/libraries with an API that could be leveraged?
