Annotating NER dataset

Question

I am working on annotating a dataset for the purpose of named entity recognition.

In principle, I have seen that for multi-phrase (not single word) elements, annotations work like this (see this example below):

1. Romania (B-CNT)
2. United States of America (B-CNT C-CNT C-CNT C-CNT)

where B-CNT stands for "beginning-country" and C-CNT represents "continuing-country".

The problem that I face is that I have a case in which (not related to countries) where I need to annotate like B-W GAP_WORD C-W C-W.

How should I proceed with the annotation in this case?

If I do annotate like in the schema above, should I expect a BERT-alike entity recognition system to learn and detect that a phrase can be like B-W GAP_WORD C-W C-W, or do I need that "C-W" (continuation word) to be exactly after the B-W (beginning word)?

Which solution is correct of the following 2:

1. B-W GAP_WORD C-W C-W
2. B-W GAP_WORD B-W C-W

And then, in case 2, find a way to make the connection between the B-Ws (actually corresponding to the same entity)?

Answer 1

As far as I'm aware there's no perfect answer to this question.

I agree with your analysis, the two options make sense:

• The first option corresponds to the correct labeling in theory, in the sense that it means exactly what one wants in this case: the words of the entity don't (necessarily) appear continuously.
• The second option makes things easier for the NER system by separating the two parts of the entity. This might lead to better results in practice, because NER systems often make errors with the B label.

If possible I would suggest to experiment with the two options. At the stage of annotation this would mean tagging such discontinuous NEs with a special temporary tag, for example:

B-W GAP_WORD B_OR_C-W C-W

this way the special B_OR_C-W can be automatically replaced with either B or C depending on the selected option.