2
$\begingroup$

Given a large set of short sentences (around 20-30 words) and multi label task (around 100 labels , can be to 3 labels per sentences ).

The location of each annotation is not impotent (i.e i only need to know if the annotation is included in the sentence)

Which method will be more beneficial ? using NER models with labels attached to the tokens from each sentence, or text classification where the sample is the whole sentence .

The labels are action that physician is doing (i.e "clean wound" , "remove skin" etc)

$\endgroup$
3
$\begingroup$

Using NER (more generally sequence labeling) means classifying every token in the sentence, so if the goal is only to label every sentence there's no strong need for it in your case.

However NER might be more appropriate in case the order of the words is important, because sequence labeling models take it into account whereas traditional text classification methods often use a "bag of words" representation (order doesn't matter).

To some extent it also depends whether the labels are always related to a particular term in the sentence: if yes, then NER might be better at locating these terms (this is related to the point about order). If no, then classifying at the level of sentences is likely to perform better.

| improve this answer | |
$\endgroup$
  • $\begingroup$ using RNN architectures (LSTM/BiLSTM) will include the order of the text into its representation in a bit.. The labels are related to text, as for the sentence "the Xray technician took a lung screen of the patient , as Dr X suggested " and the relevant label will be "lung X-ray" $\endgroup$ – Latent Feb 10 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.