1
$\begingroup$

I'm implementing a tool which should extract values of interest from unstructured text entries. The data set is several hundred thousands of medical entries. Each entry is relatively short (around 100 characters).

Each text entry contains a keyword MI (or TI or AO) and their repective value. The value is sometimes numeric and sometimes expressed as human language.

Some entries are very obvious. Some entries are more complex or ambiguous, in which case the end user of the tool (a doctor) should be able to make my tool "learn" a correct value.

Here are some examples with my comments

Data entries Expected extracted value Comments
the value of MI is 1 1.0
MI = 1 1.0
MI > 1 1.5
MI < 1 0.5
No traces of MI found 0.0
No traces of MI found 0.0
traces of MI/TI found 0.5
MI/TI/AO up to 4 3.5
MI/TI/AO up to 4 3.5
MI 2-[3] 2.5
MI probably 2. Gradient of MI 170torr 2 This example shows that a numeric value following a MI does not always represent its value. Here 170 is not the value I'm looking for. It's the 2
DOMV bez MS/MI v chlopni, ale úzký MI ?? Here I actually don't know what the value is. Only the user of the tool will be able to say what the value is in this case

Those are just some non exhaustive examples. On important aspect is that the patterns repeat often across the data set.

Another important requirement is that what we are looking must be configurable. Sometimes we look for MI sometimes for TI.

I am myself a software engineer (not a data scientist nor a doctor) so I started with a naive implementation based on regular expressions. It's basically a set of regexp matched against each entry of the data set

This kind of works and I'm able to find around 90% of the cases. However it has the following drawbacks:

  • the more regexp, the longer it will take.
  • some regexp might clash and report different values for the same entry
  • it might be cumbersome for a non tech user to add or adjust the regexps

The final user of the tool is not a software engineer but a doctor and does not know anything about regexp. However, in ambiguous/unclear cases it's the end user who can tell what the value is.

I could of course write a user interface to add/adjust/remove the rules (regexp) but that could be cumbersome as well

I was wondering if there is way without using explicit if/else rules but instead train a model which would be then used for finding the values.

I don't know much about NLP/AI or ML so maybe someone could point me to a right direction. Such as

  • Would NLP or NER be the concept to use here?
  • What could be some online services or open source tools/libraries with an API that could be leveraged?
$\endgroup$

1 Answer 1

0
$\begingroup$

One option is a data labeling system. The goal of a data labeling system is to allow less technical users to map raw data to established features. An example package is snorkel.

$\endgroup$
2
  • $\begingroup$ I was just reading about snorkel. Thanks I'll investigate. You link points to funny video about dogs though :D (probably a mistake) $\endgroup$
    – Jan
    Feb 16, 2023 at 13:50
  • $\begingroup$ Thanks for the catch! That was the wrong link. It has been updated to point to the Snorkel project page. $\endgroup$ Feb 16, 2023 at 19:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.