terminology and advice for NLP on multiclass classification with ordered levels

Question

I work in healthcare and am trying to see if I can use NLP for a classification task on complex sentences. To explain, I have different labels, and each label has multiple levels. I am not sure on the correct terminology however. I have label X, and X always exists as one of 4 'levels':

• 'absent' or 0
• 'few', or 1
• 'many', or 2
• 'everywhere', or 3

Example sentences then look like: 'I have no X', or 'I have a little bit of X', or 'there is X everywhere'. However, I also have labels Y and Z, which also have multiple levels. To complicate things further, one sentence can often contain information about multiple labels. As an example:

• 'X and Y are both absent in this man'. classification would be X:0, and Y:0
• 'He has no X but a lot of Z'. classification would be X:0 , and Z:2
• 'There is Z everywhere, but very little Y'. classification would be Z:3 and Y:1

Sentences in my corpus can also be about something completely different in which I am not interested:

• 'Q was quite small'.

Does anyone know the correct terminology for a problem like this? I have tried doing research on similar problems, I was thinking a regression based solution might be needed for the different levels, although pure multiclass classification might also work. I have quite a lot of experience with multilabel and multiclass classification problems, but without the multiple ordered 'levels' in the data I show here. Also, if anyone has any suggestions for approaches, that would also be very helpful! I do have a few thousand high quality training sentences.

Answer 1

For the target variable I would probably choose the numerical representation from 0 to 3 and consider this problem a regression task:

• Main advantage: preserve the order over the levels. I think it's really important to represent the fact that the error between levels 0 and 1 is less important than an error between levels 0 and 3.
• As a result, the predicted value would be a real number between 0 and 3 (maybe even outside this range sometimes). Typically one can re-normalize this value (replace with closest integer).
• Note that it's still reasonable enough to consider this as multiclass classification problem.

But imho the most difficult part will be the design of the NLP process, because to my knowledge this is not a standard task. Suggestions:

Approach 1

• A custom Named Entity Recognition would be trained only to detect the different target diseases. Note that if these are quite standard there can pretrained models.
• Then a different regression model would be trained for each specific disease. Maybe the input could be a short window around the target term, in order to make sure the model using the info related to this disease in case there are several.

Approach 2

• Two custom NER models, one for detecting the target diseases and another for detecting the clues related the level (e.g. 'no', 'a lot').
• Then probably a simple ad-hoc program to match the right clue to the right disease, typically based on the distance between the two.